Support for grouping packages and their deployment under their group name

[smuzaffar]

Currently packages build by bits are installed/deployed under <architecture>/<pkg_name>/<version>-<revision>. Though CMS can work around this ( need to modify few tools) but it will help if bits can add a support for grouping. e.g. all ml packages go under ml group, common packages can go under externals or common, generator packages can go under generator and cms sepecific packages can go under cms. Such a directly structure will allow use existing CMS tools to directly make use of packages build by bits. One can also set a default group so that if package itself has not provided/set it then bits can use the one from defaults.

I think this should be trivial to implement (just need to add the group in the package hash and change the package path in the INSTALL_DIR so that when it is packages it has <architecture>/<group>/<pkg_name>/<version>[-<revision>]

[pbuncic]

The problem is that what is 'common' or 'external' for one experiment is not necessarily so for others. We initially had multiple bit repositories (simulation, common, ml...). We merged them into one, and now we want to find a way to tag and separate them again. Perhaps we need to develop a 'bits deploy' tool that will take build products and copy/link them to the destination (which might include CVMFS), while adhering to rules defined in the experiment-specific configuration file.

[smuzaffar]

@pbuncic , having recipes in different repositories is different than what I have requested here. As one can copy recipes from one repo to another so having a recipe in some specific repo does not make much sense.

The group (or call it category) support is optional. If someone wants to use it then they can otherwise bits behaves as it behaves now. e.g. cms can add all its packages under cms group/category. Note that not all packages are required to have a group. If common/externals does not make any sense then there is no need to add them under any group. An experiment can define a default group (via its defaults if needed) for those packages which do not define any group/category. I have asked for this feature as CMS software deployment has been very consistent for years and many CMS project and user tools rely on this deployment structure. For CMS it would be an easier transition to bits if we provide the same deployment structure.

As I wrote, it would be a trivial change in bits to implement and avoid any tricks at actual deployment time.

[akritkbehera]

@pbuncic , having recipes in different repositories is different than what I have requested here. As one can copy recipes from one repo to another so having a recipe in some specific repo does not make much sense.

The group (or call it category) support is optional. If someone wants to use it then they can otherwise bits behaves as it behaves now. e.g. cms can add all its packages under cms group/category. Note that not all packages are required to have a group. If common/externals does not make any sense then there is no need to add them under any group. An experiment can define a default group (via its defaults if needed) for those packages which do not define any group/category. I have asked for this feature as CMS software deployment has been very consistent for years and many CMS project and user tools rely on this deployment structure. For CMS it would be an easier transition to bits if we provide the same deployment structure.

As I wrote, it would be a trivial change in bits to implement and avoid any tricks at actual deployment time.

Adding to it
The target directory will be constructed dynamically: $WORK_DIR/$ARCHITECTURE/$PACKAGE_FAMILY/.

• Global Defaults: The package_family is set to external in the release defaults.
• Package Overrides: Individual packages can specify their own package_family to override the default location.

Configuration Examples

Default Behavior:

package: defaults-release
version: v1
env:
  CXXSTD: '20'
package_family: external

Package Overrides:

package: ROOT
version: 6.36.05
package_family: lcg

ls $WORK_DIR/$ARCHITECTURE/external/ 
defaults-release autotools gcc
ls $WORK_DIR/$ARCHITECTURE/lcg/ 
ROOT

The only changes needed are the path and making this information to init generation function. Again this only have changes if you package_family otherwise it just works as before
$WORK_DIR/$ARCHITECTURE/

[pbuncic]

My point is that what you refer to as the ‘lcg’ package family in other experiments may be called differently, perhaps ‘externals’. This way, we can’t use the same recipe for both experiments. I still believe we should re-think everything in terms of a two-step process: a ‘build’ step that provides a locally consistent build with all the necessary components, ensuring that local development use cases can be fulfilled (including setting up the environment with modules, if desired), and a ‘deploy’ step where the software is deployed on CVMFS or elsewhere. In this step, additional configuration can be applied from the experiment-specific area, which could lead to package grouping or sharing of build products across releases.

[smuzaffar]

My point is that what you refer to as the ‘lcg’ package family in other experiments may be called differently, perhaps ‘externals’. This way, we can’t use the same recipe for both experiments.

totally agree with your above statement if this information goes in to the package recipe. What we are thinking is to feed in this information via the experiment defaults. So package recipe should not be changed.

I also agree that it will be a clean solution if bits break the build/packaging and deployment as separate steps. This way a pre-build package can be deployed according to the experiment needs. Such an implementation can work for both grouping and non-revisioned packages. Only technical issue I see is that there is a lot of hard-coded information saved in packages and there dependent packages (e.g. in init.sh files, module files etc.) that the deployment part will actually need to know too much about the build/packaging step. Also I guess bits should use the same deployment policy during the build/packaging phase so that experiment CI/CD can use the build (local install/deployment) area directly.