Dependent Parameters

[damskii9992]

General

It can be beneficial in a model to have Parameters which are defined through a relation to other parameters/descriptors. These dependent parameters are defined by their relation and thus should not be fitted during minimization. Their value should be updated when the values of the independent parameters which they depend on are changed. It should be possible to easily convert a dependent parameter to a dependent parameter and vice versa.

Making a Parameter dependent

There are 2 ways to make a dependent Parameter. An already created Parameter can be made dependent by using the make_dependent_on method:

a = Parameter(name='a', value=2)
b = Parameter(name='b', value=1)
b.make_dependent_on(dependency_expression='2*easypeasy', dependency_map={'easypeasy' : a})

Or by using the class method from_dependency to construct it directly:

a = Parameter(name='a', value=2)
b = Parameter.from_dependency(dependency_expression='2*easypeasy', dependency_map={'easypeasy' : a})

In either case, the Parameters internal attribute _independent will be set to False, which will lock all the setter methods and disable the parameter for fitting.
In addition, since a "fixed" and "dependent" parameter does not make sense, the fixed attribute will also be set to False.

Both the make_dependent_on and from_dependency methods take the two arguments dependency_expression and dependency_map. The dependency_expression is an arithmetic or logic expression as a string which, when evaluated, should return a Parameter or a DescriptorNumber. The dependent Parameter evaluates this expression whenever its dependencies are updated, and copies the attributes of the returned Parameter or DescriptorNumber over to itself.
The dependency_map is a dictionary mapping of string-object key-value pairs with the keys being the objects string representation in the dependency_expression and the values being a reference to the actual object.

The _independent attribute is a read-only attribute, meaning its setter simply raises a error. To make a dependent parameter independent, one has to use the make_independent() method.
This is done to ensure that the object is properly detached from the list of observers (see "The observer pattern" further down) for all its independent parameters:

def make_independent(self) -> None
   for dependency in self._dependency_map.values():
      dependency._detach_observer(self)
   self._independent = True

For the same reason, to change the dependency of an already dependent parameter, one has to use the make_dependent_on method again.

Unique names in the dependency_expression

To improve the ease of use, especially when the parameters to otherwise add to the dependency_map is hidden away behind many layers of objects, or if its destination is simply unknown, we also allow the use of unique_names in the dependency_expression:

a = Parameter(name='a', value=2, unique_name='EasyPeasy')
b = Parameter.from_dependency(dependency_expression='2*"EasyPeasy"')

To use unique_names in the dependency_expression they must be encapsulated by either "" or '' (depending on which were used to define the dependency_expression string). When unique_names are used, the objects does not need to be added to the dependency_map argument. Mixtures of dependency_map arguments and unique_names can be used.

Arithmetic and logic dependencies

To provide extremely flexible and user-friendly dependencies, we rely on the arithmetic operations between Parameters and DescriptorNumbers, as described in #54. This allows for many different kinds of complicated dependency expressions:

a = DescriptorNumber(name='a', value=2)
b = Parameter(name='b', value=10, unit='m')
c = Parameter(name='c', value=-5, unit='cm', variance=1)
d = Parameter.from_dependency(
   dependency_expression='5*a**2*b + c/a', 
   dependency_map={'a' : a, 'b' : b, 'c' : c}
   )

We also allow for logical dependencies through ternary operations, such as:

a = Parameter(name='a', value=2)
d = Parameter.from_dependency(
   dependency_expression='-a if a.value<0 else a**2'
   dependency_map={'a' : a}
   )

Implementation Details

Dependent Parameters are implemented using the generic "Observer" coding pattern, with all the dependent parameters being observers subscribing to the independent parameters.
To allow for great flexibility in the type of possible dependencies, the update of a dependent parameter is done using the asteval python interpreter with its functionality limited to only arithmetic and logical operation, and its symtable including only the independent parameters.

The observer pattern

The observer pattern is a fairly simple pattern including only 4 basic methods and 1 attribute:

class ConcreteSubject(ABC):
    """
    The Subject owns some important state and notifies observers when the state
    changes.
    """

    _state: int = None
    """
    For the sake of simplicity, the Subject's state, essential to all
    subscribers, is stored in this variable.
    """

    _observers: List[Observer] = []
    """
    List of subscribers. In real life, the list of subscribers can be stored
    more comprehensively (categorized by event type, etc.).
    """

    def _attach(self, observer: Observer) -> None:
        print("Subject: Attached an observer.")
        self._observers.append(observer)

    def _detach(self, observer: Observer) -> None:
        self._observers.remove(observer)

    """
    The subscription management methods.
    """

    def _notify_observers(self) -> None:
        """
        Trigger an update in each subscriber.
        """

        print("Subject: Notifying observers...")
        for observer in self._observers:
            observer._update(self)

This is the basic construct of the observed object, here namely the independent parameters. Since a dependent Parameter can use a DescriptorNumber for its relation, this part of the observer pattern is implemented on the DescriptorNumber

class Observer(ABC):
    """
    The Observer interface declares the update method, used by subjects.
    """

    @abstractmethod
    def _update(self, subject: Subject) -> None:
        """
        Receive update from subject.
        """
        pass

Since only Parameters can be a dependent parameter, the _update method is defined in the Parameter class.

Avoiding cyclic dependencies

The problem

Since parameters can be made dependent after they're created, it is possible to end up in an infinite loop where a dependent parameters update triggers another dependent parameters update which in turn triggers the first parameters update and so on:

flowchart LR;
   A-->|update|B;
   B-->|update|C;
   C-->|update|D;
   D-->|update|B;

Loading

The solution

To solve this we check the validity of the dependency chain whenever making a dependent parameter by calling the _validate_dependencies method (implemented in DescriptorNumber) before any of the parameters values are changed. This method pings all of the parameters observers (if any) passing along its unique_name, these parameters then ping their own observers, passing along the unique_name and so on. If the unique_names passed along the ping matches the objects own unique_name, a cyclic dependency has been detected and an error is thrown:

def _validate_dependencies(self, origin=None) -> None:
    """Ping all observers to check if any cyclic dependencies have been introduced.

    :param origin: Unique_name of the origin of this validation check. Used to avoid cyclic depenencies.
    """
    if origin == self.unique_name:
        raise RuntimeError('\n Cyclic dependency detected!\n' +
                f'An update of {self.unique_name} leads to it updating itself.\n' +
                'Please check your dependencies.')
    if origin is None:
        origin = self.unique_name
    for observer in self._observers:
        observer._validate_dependencies(origin=origin)

Only in case this dependency-traversal doesn't raise an error, is the dependent parameter updated with the dependency expression. This implementation allows for dependencies of dependencies and will detect if a cyclic dependency has been created.

The dependency_expression evaluation with asteval

When a parameter is made dependent, a python interpreter object is created with asteval and attached to it. We use asteval to limit the interpreters functionality to only arithmetic and logical/ternary expressions, to avoid many of the potential safety issues with embedded interpreters: https://lmfit.github.io/asteval/motivation.html.

The asteval interpreter has no access to the local or global namespace, so in order to use Parameters or DescriptorNumbers in it, these first have to be added to the interpreters symtable. This is the purpose of the optional argument dependency_map, which is a dictionary mapping the names used in the dependency_expression to existing objects.

After the dependency_map has been added to the asteval interpreters symtable, the dependency_expression is evaluated with it.
If the resulting output is either a DescriptorNumber or a Parameter, the dependent parameters attributes are updated to the ones of the output:

def _update(self):
   temp= self._dependency_interpreter(self._clean_dependency_string)
   self._scalar.value = temp.value
   self._scalar.unit = temp.unit
   self._scalar.variance = temp.variance
   self._min.value = temp.min if isinstance(temp, Parameter) else temp.value
   self._max.value = temp.max if isinstance(temp, Parameter) else temp.value
   self._notify_observers()

If the resulting output is not a DescriptorNumber or a Parameter, an error is raised.
Note that the min and max values are simply set to the value of the output if the output is a DescriptorNumber.

Unique names in the dependency_expression

This functionality is provided by the internal _process_dependency_unique_names method, which uses regex to scan the dependency_expression for unique_names, checks if these exists in the global_object and then adds them to the internal _dependency_map:

def _process_dependency_unique_names(self, dependency_expression: str):
   inputted_unique_names = re.findall("(\'.+?\')", dependency_expression)
   inputted_unique_names += re.findall('(\".+?\")', dependency_expression)
   for name in inputted_unique_names:
      name = name.strip("'\"")
      if name in self._global_object.map.vertices()
         parameter = self._global_object.map.get_item_by_key(name)
         self._dependency_map['__'+name+'__'] = parameter
         clean_dependency_string.replace(name, '__' +name+'__')

Note that the unique_names are pre-fixed and pre-pended with double underscores '__' in the _dependency_map and that the dependency_expression has the unique_names replaced internally with these new names instead. This is done in an attempt to avoid name clashes with names in the argument dependency_map.

Errors during the creation of a dependent parameter

If an error is thrown during the creation of a dependent Parameter, such as if the supplied dependency_expression is faulty or if a cyclic dependency is detected by the _validate_dependencies method, the Parameter is reverted to its previous state, using the data stored in the _previous_dependency dictionary and the _previous_independent boolean attributes.
This includes attaching/detaching itself as an observer if an already dependent Parameter was attempted to have its dependency_expression updated.

This functionality is provided by the _revert_dependency method which is called in the try/catch statements just before the relevant errors are raised:

def _revert_dependency(self, skip_detach=False) -> None:
    """
    Revert the dependency to the old dependency. This is used when an error is raised during setting the dependency.
    """
    if self._previous_independent is True:
        self.make_independent()
    else:
        if not skip_detach:
            for dependency in self._dependency_map.values():
                dependency._detach_observer(self)
        for key, value in self._previous_dependency.items():
            setattr(self, key, value)
        for dependency in self._dependency_map.values():
            dependency._attach_observer(self)
        del self._previous_dependency
    del self._previous_independent

Link to the ADR suggestion:
#10
Major discussions were also had during the PR here:
#112
Link to ADR suggestions changing this ADR:
#122