Upgrading from PEtab v1

Note

This guide is non-normative. It explains what the changes from PEtab v1 mean for an existing PEtab v1 problem, and how to convert such a problem to PEtab v2 so that it still describes the same estimation problem. The PEtab 2.0 specification is authoritative; its section Changes from PEtab 1.0.0 lists all changes.

Overview

Most changes between PEtab v1 and v2 are structural: the same information is written differently, and a conversion is purely mechanical. A few changes affect the numbers: the objective function, its optimum, or the prior distribution. These require a decision during the conversion. The most important one is the removal of parameterScale (Parameter scaling).

PEtab v1 never defined the objective function. It specified the likelihood, but not how priors on log-scaled parameters enter the objective, and tools implemented this differently. PEtab v2 defines the objective function completely. In this guide, a conversion is called faithful if the v2 problem has the same likelihood and the same optimum as the v1 problem, and, for problems with priors, the same prior distribution and the same objective function up to an additive constant. “The same” refers to how the v1 problem was evaluated by the tools that created it (How PEtab v1 tools implemented it).

Quick path

For a typical v1 problem (SBML model, parameters estimated on log10 scale, few or no priors):

Convert with the libpetab-python function petab.v2.petab1to2:
```
from petab.v2 import petab1to2

petab1to2("v1/problem.yaml", output_dir="v2", reparametrize="all")
```
Pass reparametrize="all" explicitly. The default, "none", keeps all parameters on linear scale and moves the optimum for parameters with parameterScale* priors. With "all", every estimated log10-scaled parameter k is replaced by a parameter log10_k with log-transformed bounds, nominal value and prior, and the model computes k = 10^log10_k (log_k and exp(log_k) for log scale). Parameters with a linear-scale prior (normal, laplace, logNormal, logLaplace, uniform) and parameters with a lower bound of 0 stay on linear scale. The options are compared in With libpetab-python. In short:
- No priors: "all" keeps gradients with respect to the log-transformed value, optimization scale and start point sampling as in v1. "none" gives the same optimum, but the parameters are then estimated on linear scale.
- parameterScale* priors: "all" (or "priors"). "none" moves the optimum.
- Only linear-scale priors: these parameters are not reparametrized by any option.
Read the warnings and do what petab1to2 does not do:
- Rewrite observableFormula expressions that reference other observables (Changes without numerical consequences).
- If the conversion fails because a measurement table placeholder receives overrides that cannot be log-transformed, split the affected observable first (Reparametrizing a parameter).
- Configure non-default initialization priors in the tool (Initialization priors).
Adjust the tool settings. Optimization scale and start point sampling are no longer part of the problem. The reparametrized parameters are already log-transformed and must not be log-transformed again by the tool.
Check the result as described in Verifying a conversion.

The rest of this guide explains the individual changes and how to perform a conversion by hand.

Notation

As in the v2 specification, $\theta$ denotes the vector of parameters as they appear in the parameter table, and $x$ a single such parameter. In PEtab v1, these are the values on linear scale, which is what lowerBound, upperBound and nominalValue refer to. $\tilde{x}$ denotes the value of a v1 parameter on its parameterScale, i.e., $\tilde{x} = \log_{10} x$ for log10, $\tilde{x} = \ln x$ for log, and $\tilde{x} = x$ for lin. $\tilde{\theta}$ is the respective vector. For example, a log10-scaled parameter with nominalValue 100 has $x = 100$ and $\tilde{x} = 2$. A parameter with parameterScale log or log10 is called log-scaled below. Formulas are written for log10; for log, $\ln 10$ becomes $1$, and log10_k with k = 10^log10_k becomes log_k with k = exp(log_k).

Changes without numerical consequences

The following changes only affect how a problem is written. petab1to2 handles all of them, except where noted.

PEtab v1	PEtab v2	Notes, manual steps
Optional YAML file with a `problems` list	Mandatory YAML file with a flat structure and `model_files` entries with `location` and `language`
Condition table in wide format, one column per overridden entity	Condition table in long format, one row per `conditionId`, `targetId` and `targetValue`	Conditions that don’t change anything are not needed anymore. The condition table is optional.
`simulationConditionId` and `preequilibrationConditionId` in the measurement table	`experimentId` in the measurement table and an experiment table	A pre-equilibration condition becomes a period starting at `-inf`, the simulation condition a period starting at `0`. Measurements without any condition don’t need an experiment.
`observableParameter${n}_${observableId}` and `noiseParameter${n}_${observableId}` placeholders by naming convention	Placeholders declared in the `observablePlaceholders` and `noisePlaceholders` columns of the observable table	The v1 names remain valid placeholder IDs.
`observableTransformation` (`lin`, `log`, `log10`) and `noiseDistribution` (`normal`, `laplace`)	A single `noiseDistribution` column (`normal`, `log-normal`, `laplace`, `log-laplace`)	`log10` requires a rescaled noise formula, which `petab1to2` inserts, see log10-transformed observables.
`estimate` as `0` / `1`	`estimate` as `false` / `true`
`objectivePriorType` and `objectivePriorParameters`	`priorDistribution` and `priorParameters`	The set of distributions changed. `logNormal` becomes `log-normal`, `logLaplace` becomes `log-laplace`. For the `parameterScale*` priors, see Parameter scaling.
Visualization table	Removed	Visualization files are not part of a PEtab v2 problem. libpetab-python documents its own visualization input format.
Observable IDs allowed in `observableFormula`	Not allowed; only an observable’s own ID may be used in its `noiseFormula`	By hand: substitute the referenced observable’s formula. `petab1to2` does not do this.

Parameter scaling

In short: PEtab v2 has no parameterScale. A parameter k that was estimated on log10 scale in v1 is still estimated on log10 scale in v2 by making the log-transformed value the parameter: log10_k is listed in the parameter table, and the model computes k = 10^log10_k. For a faithful conversion, this reparametrization is recommended for all log- and log10-scaled parameters, except those with a linear-scale prior (normal, laplace, logNormal, logLaplace, uniform). The steps are described in Reparametrizing a parameter; petab1to2 performs them with reparametrize="all" (With libpetab-python). The following subsections explain why.

What PEtab v1 specified

In PEtab v1, every parameter had a parameterScale of lin, log or log10, the “scale of the parameter to be used during parameter estimation”. Bounds and nominal values were always given on linear scale. Priors came in two groups:

uniform, normal, laplace, logNormal and logLaplace were defined “on linear parameters”, with prior parameters on linear scale.
parameterScaleUniform, parameterScaleNormal and parameterScaleLaplace were defined “on original parameter scale”, with prior parameters on parameter scale. For example, parameterScaleNormal with parameters mu;sigma on a log10-scaled parameter means $\log_{10} x \sim \mathcal{N}(\mu, \sigma^2)$.
The default was parameterScaleUniform within the bounds, described as “equivalent to no prior”.

In addition, initializationPriorType specified how start points for optimization were sampled, with the same default. This is how parameterScaleUniform originally came about (PEtab#402).

Two things were not specified. First, the objective function itself. Second, and as a consequence, how a prior on a log-scaled parameter enters it: as a density over the scaled parameter $\tilde{x}$, or as a density over $x$. The two differ by the Jacobian of the scale transformation, which adds $\ln(x \ln 10)$ to the negative log-prior for log10. This term is not constant, so the two readings have different optima.

How PEtab v1 tools implemented it

The following describes the behavior of some tools at the linked commits. Other tools have not been surveyed.

D2D and pyPESTO evaluate parameterScale* priors as densities over the scaled parameter $\tilde{x}$, without a Jacobian (D2D arCollectRes.m, pyPESTO priors.py). pyPESTO evaluates the linear-scale priors (normal, logNormal, …) as densities over $x$, also without a Jacobian. Problems that were created with these tools have their published optima under this reading. This was confirmed for the problems with priors in the benchmark collection.
libpetab-python’s v1 Prior class evaluates all priors as densities over $x$, with the Jacobian (priors.py). For a parameterScaleNormal prior on a log10-scaled parameter, this is a log-normal density over $x$, and the optimum differs from the D2D/pyPESTO one.
Gradients are commonly reported with respect to the scaled parameters $\tilde{\theta}$ (e.g., AMICI, pyPESTO), and start points are sampled on the parameter scale. Optimizer steps and convergence criteria depend on which variable the gradient refers to.

For log-scaled parameters with priors, objective values and optima therefore differ between v1 tools. Gradients with respect to $\tilde{x}$ and with respect to $x$ differ by the factor $x \ln 10$ for a log10-scaled parameter. This guide follows the D2D/pyPESTO reading.

What PEtab v2 specifies

PEtab v2 has no parameterScale. The parameters in the parameter table are the estimated parameters; bounds, nominal values and priors all refer to them. Priors are densities over the parameter as it appears in the table, and the objective function is fully defined. How a tool optimizes or samples internally is not part of the problem.

Estimating a parameter on log10 scale is still possible, by making the log-transformed value the parameter: a new parameter log10_k is estimated, and the model computes k = 10^log10_k. This is what the specification refers to with “the model parameters can be transformed in the model file”.

Consequences for a conversion

Parameters without a prior. The likelihood does not depend on how a parameter is parametrized. Keeping $x$ as the estimated parameter changes nothing numerically. What changes is the meaning of the gradient (now with respect to $x$ instead of $\tilde{x}$), and that optimization scale and start point sampling become tool settings. Reparametrizing to $\tilde{x}$ keeps all of these as they were. The v1 default prior (flat in $\tilde{x}$) is then also kept, see Default prior.

Parameters with a parameterScale* prior. The v1 prior is a density over $\tilde{x}$. A v2 prior is a density over the estimated parameter. If $x$ stays the estimated parameter, no v2 prior preserves both the prior distribution and the optimum. The following table compares the possible v2 encodings for the example of a parameterScaleNormal prior with parameters mu;sigma on a log10-scaled parameter:

v2 encoding of `parameterScaleNormal(mu, sigma)` on `log10` scale	Prior distribution	Optimum and objective	Gradient with respect to
`log-normal(muln10, sigmaln10)` on $x$	Preserved	The objective differs by $\ln(x \ln 10)$; the maximum a posteriori (MAP) estimate of $\log_{10} x$ moves by $-\sigma^2 \ln 10$	$x$ instead of $\tilde{x}$
`log-normal` on $x$ with a shifted mean, `(mu + sigma^2ln10)ln10`	Changed; the prior no longer states the intended belief	Objective preserved up to a constant	$x$ instead of $\tilde{x}$
`normal(mu, sigma)` on the new parameter $\tilde{x}$ (reparametrization)	Preserved	Preserved, up to the constant from truncation	$\tilde{x}$, as in v1

The MAP estimate moves because the maximum of a density is not invariant under a change of variables. For $\sigma \le 0.2$ the shift is below 0.1 in $\log_{10} x$, but for wide priors such as $\sigma = 3$ it is 20, i.e., onto a bound.

Parameters with a linear-scale prior (normal, laplace, logNormal, logLaplace, uniform). Under the pyPESTO reading, the prior is a density over $x$. The corresponding v2 prior on $x$ gives the same objective. These parameters are therefore not reparametrized. (Reparametrizing them with the corresponding prior on $\tilde{x}$, e.g., log-normal to normal, keeps the distribution but moves the optimum, by the same argument as above.)

Recommendation. Estimate the variable over which the v1 prior was a density: $\tilde{x}$ for parameterScale* priors, $x$ for linear-scale priors. Parameters without a prior can be handled either way. Reparametrizing them to $\tilde{x}$ keeps gradients with respect to $\tilde{x}$, start point sampling and default prior as in v1, and is recommended for a faithful conversion. This corresponds to reparametrize="all" in petab1to2.

Reparametrizing a parameter

The following describes how to replace a log10-scaled parameter k by log10_k, with k = 10^log10_k. For log scale, use log_k and exp(log_k).

Parameter table. Replace the row for k by a row for log10_k with the log-transformed bounds and nominal value. For example:

	`parameterId`	`lowerBound`	`upperBound`	`nominalValue`	`estimate`	`parameterScale`
v1	`k_phos`	`1e-5`	`1e3`	`100`	`1`	`log10`
v2	`log10_k_phos`	`-5`	`3`	`2`	`true`	(removed)

The prior, if any, becomes a prior on log10_k:

v1 prior on `k` (`log10` scale)	v2 prior on `log10_k`
`parameterScaleNormal` `mu;sigma`	`normal` `mu;sigma`
`parameterScaleLaplace` `mu;sigma`	`laplace` `mu;sigma`
`parameterScaleUniform` `a;b`	`uniform` `a;b`
none (v1 default)	none, or `uniform` `log10(lb);log10(ub)` (see Default prior)

A parameter with a lower bound of 0 cannot be reparametrized this way, since its bound would become -inf. PEtab v1 allowed this for log-scaled parameters with an initialization prior. Such a parameter stays on linear scale, which is what petab1to2 does; reparametrizing it anyway requires choosing a positive lower bound, which changes the problem.

Model. If k is a model parameter, add a constant parameter log10_k to the model and define k by an SBML initial assignment k = 10^log10_k, replacing any existing initial assignment for k. k stays a constant parameter and can still be used as a condition target. For the example above:

<parameter id="log10_k_phos" value="2" constant="true"/>
<parameter id="k_phos" value="100" constant="true"/>
<initialAssignment symbol="k_phos">
  <math xmlns="http://www.w3.org/1998/Math/MathML">
    <apply><power/><cn>10</cn><ci>log10_k_phos</ci></apply>
  </math>
</initialAssignment>

Alternatives:

An assignment rule k = 10^log10_k also works, but makes k a non-constant entity. It can then no longer be the target of a condition or of an event assignment, and must not appear in the parameter table. Its only advantage is robustness in tools that don’t re-evaluate initial assignments after parameter changes.
Substituting 10^log10_k for k in all model expressions and removing k changes every kinetic law in which k occurs and loses its annotations.
Defining k in the condition table for every experiment spreads the model definition over the experiments.

Observable and noise formulas, condition table. Replace k by 10^log10_k in observableFormula, noiseFormula and targetValue expressions. This is required if k is not a model parameter, and otherwise harmless, as the model already defines k.

Measurement table overrides. Values in observableParameters and noiseParameters cannot be expressions, so 10^log10_k cannot be substituted there. Instead, transform the placeholder: replace the placeholder ph by 10^ph in the observable or noise formula, and log-transform all overrides of that placeholder: k becomes log10_k, and a number c becomes log10(c). For example, an estimated noise parameter sd on log10 scale:

	v1	v2
Observable table	`noiseFormula`: `noiseParameter1_obs`	`noiseFormula`: `10^noiseParameter1_obs`; `noisePlaceholders`: `noiseParameter1_obs`
Measurement table	`noiseParameters`: `sd`	`noiseParameters`: `log10_sd`
Parameter table	`sd`, bounds `1e-5` to `1e5`	`log10_sd`, bounds `-5` to `5`

This requires that all overrides of that placeholder can be log-transformed: reparametrized parameters or positive numbers. If the same placeholder also receives non-positive numbers, or parameters that are not reparametrized, split the observable into copies, such that each copy’s placeholder only receives overrides that can be log-transformed, or none at all. Alternatively, leave the affected parameters on linear scale; this is only an option for parameters without a parameterScale* prior.

With libpetab-python

petab1to2 applies the above through its reparametrize argument:

"none" (default): no reparametrization. All parameters are estimated on linear scale. parameterScaleNormal and parameterScaleLaplace priors on log-scaled parameters are converted to log-normal and log-laplace priors on $x$, which preserves the distribution but moves the optimum, as described above. The v1 default prior becomes uniform within the bounds. This is appropriate for problems without priors, if gradients with respect to $x$ instead of $\tilde{x}$ and estimation on linear scale are acceptable.
"priors": reparametrizes the estimated log-scaled parameters with parameterScale* priors, and any log-scaled parameters that share a measurement table placeholder with them.
"all": reparametrizes all estimated log-scaled parameters, except those with linear-scale priors. Recommended for a faithful conversion.

Parameters with a lower bound of 0 are left on linear scale, with a warning. The same applies to parameters without a parameterScale* prior whose placeholder cannot be log-transformed. If a parameter with such a prior shares a placeholder with overrides that cannot be log-transformed, the conversion fails, and the affected observable needs to be split by hand first, as described above.

The underlying function petab.v2.reparametrize_log can also be applied to an existing v2 problem:

import petab.v2

problem = petab.v2.Problem.from_yaml("v2/problem.yaml")
petab.v2.reparametrize_log(problem, ["k_phos", "sd_pSTAT5A_rel"], base=10)
problem.to_files("v2_log/")

Other changes with numerical consequences

`log10`-transformed observables

PEtab v2 has no log10 noise distributions. The v1 log10-normal distribution of $m$ with noise parameter $\sigma$ is the v2 log-normal distribution with noise parameter $\sigma \ln 10$, with identical density, since $\ln m = \ln 10 \cdot \log_{10} m$. The same holds for log10-Laplace and log-laplace.

To keep estimated noise parameters and noise overrides in the measurement table unchanged, put the factor into the noise formula: noiseFormula sigma becomes ln(10) * (sigma). petab1to2 does this and warns about it. Parameters in such noise formulas keep their v1 meaning: they refer to the log10-transformed observable.

Initialization priors

initializationPriorType and initializationPriorParameters are removed. How start points are sampled is a tool setting. The v1 default, uniform on parameter scale within the bounds, is reproduced by sampling the reparametrized parameters uniformly within their (log-transformed) bounds. Non-default initialization priors have no v2 counterpart and must be configured in the tool.

Bounds truncate priors

In PEtab v2, a prior with support beyond the parameter bounds is truncated and renormalized. This adds the constant $-\ln Z$ to the objective, with $Z = \text{CDF}(\text{upperBound}) - \text{CDF}(\text{lowerBound})$ of the non-truncated prior. PEtab v1 did not specify this, and tools did not necessarily renormalize. The optimum is unaffected. For priors that are narrow compared to the bounds, $Z \approx 1$; for wide priors, the constant can be of order 1 per prior.

Default prior

In PEtab v2, if any parameter has a prior, all other estimated parameters default to uniform within their bounds, which contributes $\ln(\text{upperBound} - \text{lowerBound})$ per parameter. The v1 default parameterScaleUniform contributed nothing. The constant does not affect the optimum, but it does depend on the bounds, which matters when objective values are compared between models with different bounds or different numbers of parameters.

As a prior distribution, the v2 default is uniform in the estimated parameter. For a parameter that was log10-scaled in v1 and is not reparametrized, with bounds such as 1e-5 to 1e3, this puts most of the probability mass in the top decade, while the v1 default was uniform in $\log_{10} x$. For reparametrized parameters, the v2 default is uniform in log10_k, as in v1. This matters for Bayesian inference, not for optimization.

Summary of objective constants

After a conversion with reparametrize="all", the v2 objective differs from the v1 objective in the D2D/pyPESTO reading only by additive constants:

$-\ln Z_i$ for every truncated prior $i$;
$\ln(\text{ub}_j - \text{lb}_j)$ for every estimated parameter $j$ without an explicit prior, if any parameter has a prior;
the normalization constants of the priors and noise distributions that a v1 tool may or may not have included.

Without reparametrization, the objective additionally differs by the non-constant Jacobian term for every log-scaled parameter with a parameterScale* prior, $\ln(x \ln 10)$ for log10 scale.

Verifying a conversion

A converted problem should be checked against the original with the tools that were used with the v1 problem:

Validate both problems (petablint).
Simulate both at the nominal parameters. The simulated values for all measurements must agree to solver tolerance.
Compare the negative log-likelihood. It must agree to solver tolerance; there is no constant from the conversion.
Compare the log-prior, evaluated for v1 as a density over $\tilde{\theta}$. The difference must be the constant from the truncation and default priors above.
Compare the gradient. For reparametrized parameters, the v1 gradient with respect to $\tilde{x}$ equals the v2 gradient with respect to log10_k. For parameters that were log10-scaled in v1 and are not reparametrized, the v1 gradient equals the v2 gradient with respect to $x$ times $x \ln 10$. For v1 lin parameters, the gradients agree directly.
Repeat steps 2 to 5 at a second parameter vector, e.g., moved some way towards the center of the bounds, so that the gradient is not zero and the comparison is not specific to the optimum.

For an example, see the conversion of Schwen_PONE2014 in the benchmark collection (Benchmark-Models-PEtab#336), a problem with parameterScaleNormal priors and log10-transformed observables.