add instrument weights to global bootstrap

[pcaspers]

resolves #2396

Question:s How backwards compatible do we want to be, i.e.

• can we maybe incorporate the scale into the quoteError()?
• do we want an option to enable the scaling in IterativeBootstrap (maybe yes, because it is widely used and we might change many results) and in GlobalBootstrap (maybe not necessary, since it is probably not so commonly used)

[coveralls]

coverage: 74.359% (+0.003%) from 74.356%
when pulling 3c0a679 on pcaspers:ratehelper_scale
into 4f07325 on lballabio:master.

[eltoder]

I think this is not the right way to go about this.

In general, for global bootstrap we can assign weights to benchmarks. There are many ways to come up with weights, generally based on liquidity and "importance", so weights should be provided by the user. This is how we deal with futures in our code -- we scale their weights by 1e-2. Currently, global bootstrap does not allow passing custom weights -- all quote errors get a weight of 1. (Partially because of this we pass an empty list of helpers into the curve and add everything via additionalHelpers and additionalPenalties.) I think it would be good to allow passing custom weights into the bootstrapper. This is a generic and backwards compatible change that gives a lot of flexibility. You can also use it to solve your issue with normalizing futures vs swaps.

If we think that this is not good enough and want a more out-of-the-box experience, I think we should change quoteError(). We could just add a scale to the base class and multiply the difference inside of quoteError() like you are suggesting in the comment. But a more generic thing to do is to make quoteError virtual. Then we can override it for futures to just return values in the rates space directly. I think this is cleaner.

Other notes on your change:

1. I agree that the scale for futures is 0.01, but not sure why are you adding 100 for swaps? Swaps are already in the rates space, so should just be 1. Also, you missed another constructor and other types of swaps (at least OIS, but probably others too).
2. For the iterative bootstrap the scale of errors does not make a difference, because the accuracy is for the x values (dfs or rates), not for the f values (errors).

[pcaspers]

Thanks for your feedback @eltoder. The changes were not meant to be complete. I just wanted to sketch one possible approach to start the discussion.

You are right, what I did is not consistent, I wanted to scale all quote errors to order 1, i.e., to scale the swap quote error by 1E2, but leave the future quote error untouched.

Anyway, I agree that user-provided weights are more flexible, and I think we don't need to normalize the orders of the quote errors then. I can make this change in this branch here.

And agreed on the IterativeBootstrap, this is independent of the scale of the errors.

[eltoder]

If you want to scale swaps, you'll also need to scale deposits, fras, etc? I think there are fewer helpers for futures (just regular and overnight), so scaling futures seems easier. But in any case, if you go with custom weights in global bootstrap, everyone can do their own thing.

[pcaspers]

Yes, indeed. The custom weights are a much better approach.

[lballabio]

How do weights work with this sort? If we didn't have transparency on this implementation, it would seem natural to pass a vector of weights corresponding to the vector of instruments passed to the curve (as in, weights[i] corresponds to helpers[i]), but this seems to require that the weights correspond to the instruments after they go through sorting (and if that's the case, we'd need std::stable_sort here or we wouldn't be able to predict the order if two maturities are the same).

[pcaspers]

Oops, good catch. I'll fix it. The sorting of the weights should correspond to the original instrument sorting

[pcaspers]

I removed the sorting, I do not think it is necessary for the global bootstrap.

While reading through the code I noticed a potential issue with the underlying interpolation: This should cover the range from t = 0 to the maximum of the last relevant dates of the given helpers. However, the times vector used to build the interpolation contains the pillar dates of the helpers, which might be earlier. I guess we should enable extrapolation on the interpolation instance therefore to cover the general case?

I am slightly confused about this point, since the same issue seems to apply to IterativeBootstrap, i.e., how can we retrieve the quote error from a helper if its pillar date is earlier than its last relevant date?

[pcaspers]

Please ignore my last comment. Whether extrapolation is allowed is not controlled by the interpolation object, but by the TermStructure and maxDate(), maxTime(). maxDate_ is set explicitly to the maximum of all last relevant dates. In addition, the extrapolation does not use the interpolator anyway, but assumes a flat instantaneous forward rate.

So I'll do something similar in the global bootstrap.

[pcaspers]

In effect, we fix the case where the additional dates do not cover the requirements of the additional helpers w.r.t. their latest relevant dates.