From 4e588d5121368998f2e1da556b36010dea7a16c6 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Mon, 1 Dec 2025 19:45:36 +0100 Subject: [PATCH 01/19] [FEAT] modernization plan --- MODERNIZATION_PLAN.md | 751 +++++++++++++++++++++++++++++++++++++++++ README.md | 30 ++ UPDATING_FREESURFER.md | 21 ++ 3 files changed, 802 insertions(+) create mode 100644 MODERNIZATION_PLAN.md create mode 100644 UPDATING_FREESURFER.md diff --git a/MODERNIZATION_PLAN.md b/MODERNIZATION_PLAN.md new file mode 100644 index 0000000..e1c03a4 --- /dev/null +++ b/MODERNIZATION_PLAN.md @@ -0,0 +1,751 @@ +# Service Modernization Plan: JupyterLab Medical Image Processing + +**Date:** December 1, 2025 +**Service Version:** 1.2.1 +**Maintainer:** ordonez + +--- + +## Executive Summary + +This document outlines a comprehensive plan to modernize the JupyterLab Medical Image Processing (MedImProc) service. The modernization focuses on documenting current software versions, updating to newer releases, optimizing build times, and reducing image size through improved Dockerfile practices. + +--- + +## 1. Current Status Documentation + +### 1.1 Base Image +- **Base:** `itisfoundation/jupyter-math:2.0.9` +- **Status:** Inherited from upstream, requires investigation +- **How to verify:** Check Docker Hub or source repository for jupyter-math + +### 1.2 Medical Image Processing Software Versions + +#### MRtrix3 +- **Current Version:** 3.0.4 (Git checkout) +- **Installation Method:** Compiled from source (GitHub) +- **Repository:** https://github.com/MRtrix3/mrtrix3.git +- **How to check version:** + ```bash + docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 /bin/bash -c "mrinfo --version" + ``` +- **Location in Dockerfile:** Lines 23-27 + +#### FreeSurfer +- **Current Version:** 6.0.0 +- **Installation Method:** Pre-compiled binary (FTP download) +- **Download URL:** ftp://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/6.0.0/freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.0.tar.gz +- **How to check version:** + ```bash + docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 /bin/bash -c "source ${FREESURFER_HOME}/SetUpFreeSurfer.sh && freesurfer --version" + ``` +- **Location in Dockerfile:** Lines 41-49 +- **Note:** Requires license file (freesurfer_license.txt) + +#### FSL +- **Current Version:** Unknown (downloaded via installer.py) +- **Installation Method:** Python installer script +- **Download URL:** https://fsl.fmrib.ox.ac.uk/fsldownloads/fslinstaller.py +- **How to check version:** + ```bash + docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 /bin/bash -c "cat ${FSLDIR}/etc/fslversion" + ``` +- **Location in Dockerfile:** Lines 57-63 +- **Issue:** Version not explicitly pinned, downloads latest available at build time + +#### ANTs (for Synb0-DISCO) +- **Current Version:** 2.4.4 +- **Installation Method:** Pre-compiled binary (GitHub releases) +- **Download URL:** https://github.com/ANTsX/ANTs/releases/download/v2.4.4/ants-2.4.4-ubuntu-20.04-X64-gcc.zip +- **How to check version:** + ```bash + docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 antsRegistration --version + ``` +- **Location in Dockerfile:** Lines 105-107 + +#### c3d (Convert3D) +- **Current Version:** 1.0.0 +- **Installation Method:** Pre-compiled binary (SourceForge) +- **Download URL:** https://sourceforge.net/projects/c3d/files/c3d/1.0.0/c3d-1.0.0-Linux-x86_64.tar.gz +- **How to check version:** + ```bash + docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 c3d -version + ``` +- **Location in Dockerfile:** Line 109 + +#### Synb0-DISCO +- **Current Version:** master branch (unpinned) +- **Installation Method:** Git clone from GitHub +- **Repository:** https://github.com/MASILab/Synb0-DISCO +- **How to check version:** + ```bash + docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 /bin/bash -c "cd ${HOME}/synb0-disco && git log -1 --format='%H %ci'" + ``` +- **Location in Dockerfile:** Lines 75-103 +- **Issue:** Not pinned to specific version, uses master branch +- **Dependencies:** PyTorch (CPU version), installed via pip + +#### Spinal Cord Toolbox (SCT) +- **Current Version:** 4.2.1 +- **Status:** TEMPORARILY REMOVED (commented out in Dockerfile) +- **Reason:** GitHub building space issues (see lines 112-115) +- **Installation Method:** When enabled, downloads from GitHub releases +- **Download URL:** https://github.com/neuropoly/spinalcordtoolbox/archive/4.2.1.tar.gz +- **Note:** Adds ~1.5GB to image + +#### ART ACPCdetect (for MRtrix3) +- **Current Version:** 2.0 (minified) +- **Installation Method:** Pre-compiled binary (OSF download) +- **Download URL:** https://osf.io/73h5s/download +- **Location in Dockerfile:** Lines 29-31 + +#### ANTs for MRtrix3 +- **Current Version:** 2.3.4-2 (minified) +- **Installation Method:** Pre-compiled binary (OSF download) +- **Download URL:** https://osf.io/yswa4/download +- **Location in Dockerfile:** Lines 33-35 +- **Note:** Different ANTs version than the one used for Synb0-DISCO + +### 1.3 Python Packages +Defined in `requirements.in`: +- **nibabel** - Version: Unknown (not pinned) +- **pyvista** - Version: Unknown (not pinned) +- **PyOpenGL** - Version: Unknown (not pinned) +- **PyOpenGL_accelerate** - Version: Unknown (not pinned) +- **fsleyes** - Version: Unknown (not pinned) +- **connected-components-3d** - Version: Unknown (not pinned) +- **wxpython** - Installed separately with Ubuntu 20.04 wheels +- **attrdict** - Installed separately +- **PyTorch, torchvision, torchaudio** - CPU version (not in requirements.in) + +**How to check versions:** +```bash +docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 /bin/bash -c "source .venv/bin/activate && pip list" +``` + +**Generated file:** `requirements.txt` is compiled from `requirements.in` using pip-tools during build + +### 1.4 System Dependencies +- Base OS: Ubuntu 20.04 (inherited from jupyter-math base image) +- Key libraries: + - Qt5 (for MRtrix3 GUI components) + - FFTW3 (for MRtrix3) + - Eigen3 (for MRtrix3) + - OpenGL/Mesa libraries + - Various system utilities (curl, git, jq, etc.) + +--- + +## 2. Version Investigation Plan + +### 2.1 Priority 1: Critical Software (High Impact) + +#### FSL +- **Action:** Determine current installed version +- **Method:** + 1. Build current image + 2. Run: `cat ${FSLDIR}/etc/fslversion` + 3. Check FSL website for latest version: https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FslInstallation +- **Latest Known:** FSL 6.0.7.x (as of 2024) +- **Update Strategy:** Pin to specific version by modifying installer script or using direct download URL + +#### MRtrix3 +- **Current:** 3.0.4 +- **Latest Check:** https://github.com/MRtrix3/mrtrix3/releases +- **Latest Known:** 3.0.4 is latest stable release (as of check date) +- **Action:** Verify if any newer patches or releases are available + +#### PyTorch +- **Current:** CPU version, unpinned +- **Action:** Pin to specific version +- **Latest Check:** https://pytorch.org/get-started/previous-versions/ +- **Recommendation:** Use latest stable CPU-only version compatible with Python version in base image + +### 2.2 Priority 2: Moderate Updates + +#### FreeSurfer +- **Current:** 6.0.0 (from 2016) +- **Latest Check:** https://surfer.nmr.mgh.harvard.edu/fswiki/DownloadAndInstall +- **Latest Known:** FreeSurfer 7.4.x (as of 2024) +- **Consideration:** Major version update may require significant testing +- **Action:** Test FreeSurfer 7.x compatibility with existing pipelines + +#### ANTs +- **Current:** 2.4.4 (for Synb0), 2.3.4-2 (for MRtrix3) +- **Latest Check:** https://github.com/ANTsX/ANTs/releases +- **Latest Known:** 2.5.x +- **Issue:** Two different versions installed for different purposes +- **Action:** Consolidate to single ANTs installation if possible + +#### Synb0-DISCO +- **Current:** master branch (unpinned) +- **Action:** Pin to specific commit hash or tag +- **Latest Check:** https://github.com/MASILab/Synb0-DISCO/releases +- **Recommendation:** Use latest tagged release or pin commit hash + +### 2.3 Priority 3: Python Packages + +#### All Python packages in requirements.in +- **Action:** Create `requirements-versions.txt` with pinned versions +- **Method:** + 1. Build current image + 2. Export: `pip freeze > requirements-pinned.txt` + 3. Research latest versions on PyPI + 4. Test compatibility before updating + +#### Spinal Cord Toolbox +- **Current:** 4.2.1 (DISABLED) +- **Latest Check:** https://github.com/neuropoly/spinalcordtoolbox/releases +- **Latest Known:** 6.x +- **Action:** Evaluate if re-enabling is needed; if so, update to latest version + +--- + +## 3. Dockerfile Optimization Strategy + +### 3.1 Multi-Stage Build Implementation + +**Current Status:** +- Single-stage Dockerfile (current production) +- Incomplete multi-stage Dockerfile exists (MultiStage_Dockerfile) +- MultiStage_Dockerfile appears to be work-in-progress + +**Proposed Multi-Stage Architecture:** + +```dockerfile +# Stage 1: Base builder image with common build tools +FROM buildpack-deps:jammy AS base-builder + +# Stage 2: MRtrix3 builder +FROM base-builder AS mrtrix3-builder +# Compile MRtrix3 with dependencies + +# Stage 3: FreeSurfer downloader +FROM base-builder AS freesurfer-installer +# Download and extract FreeSurfer + +# Stage 4: FSL installer +FROM base-builder AS fsl-installer +# Install FSL + +# Stage 5: Synb0-DISCO setup +FROM base-builder AS synb0-installer +# Clone and setup Synb0-DISCO + +# Stage 6: Python environment builder +FROM base AS python-builder +# Install Python packages in venv + +# Stage 7: Final production image +FROM itisfoundation/jupyter-math:2.0.9 AS production +# Copy only necessary artifacts from previous stages +``` + +**Benefits:** +- Parallel builds for independent components +- Smaller final image (no build tools) +- Better layer caching +- Easier to maintain and update individual components + +### 3.2 Layer Optimization Strategies + +#### A. Combine RUN Commands +**Current Problem:** Multiple separate RUN commands create unnecessary layers + +**Solution:** +```dockerfile +# Before (multiple layers) +RUN apt-get update +RUN apt-get install -y package1 +RUN apt-get install -y package2 + +# After (single layer) +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + package1 \ + package2 && \ + rm -rf /var/lib/apt/lists/* +``` + +#### B. Minimize Package Installation +- Use `--no-install-recommends` flag +- Remove package manager cache in same layer +- Audit installed packages for necessity + +#### C. Optimize File Operations +- Download and extract in single RUN command +- Clean up temporary files in same layer +- Remove .git directories after clone + +### 3.3 Image Size Reduction Tactics + +#### Current Issues: +1. **FreeSurfer subjects directory:** Kept for fsaverage (needed for recon-all) +2. **Multiple ANTs installations:** 2.3.4-2 and 2.4.4 both present +3. **Build artifacts:** Some tools compiled from source leave build files +4. **Uncompressed downloads:** Large tarballs downloaded and not cleaned up +5. **Git history:** Full git clones with history + +#### Proposed Solutions: + +**1. FreeSurfer Optimization:** +```dockerfile +# Keep only essential subjects (fsaverage) and remove others +RUN rm -rf ${FREESURFER_HOME}/subjects/cvs_avg35* \ + ${FREESURFER_HOME}/subjects/bert \ + ${FREESURFER_HOME}/trctrain +``` + +**2. Consolidate ANTs:** +```dockerfile +# Use single ANTs version for all purposes +# Test compatibility first +``` + +**3. Shallow Git Clones:** +```dockerfile +# Before +RUN git clone https://github.com/MRtrix3/mrtrix3.git + +# After +RUN git clone --depth 1 --branch 3.0.4 https://github.com/MRtrix3/mrtrix3.git && \ + cd mrtrix3 && rm -rf .git +``` + +**4. Cleanup Build Dependencies:** +```dockerfile +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + build-essential \ + git \ + # ... build dependencies ... && \ + # ... build software ... && \ + apt-get remove -y build-essential git && \ + apt-get autoremove -y && \ + rm -rf /var/lib/apt/lists/* +``` + +### 3.4 Build Time Optimization + +#### A. Use BuildKit +```dockerfile +# syntax=docker/dockerfile:1.4 +# Enable BuildKit features +``` + +**Features to leverage:** +- `RUN --mount=type=cache` for package manager caches +- `RUN --mount=type=bind` for temporary file access +- Parallel stage execution + +#### B. Optimize Download Steps +```dockerfile +# Use --link flag for COPY (BuildKit) +COPY --link --from=builder /app /app + +# Cache pip downloads +RUN --mount=type=cache,target=/root/.cache/pip \ + pip install -r requirements.txt +``` + +#### C. Order Dockerfile for Cache Efficiency +- Put frequently changing commands last +- Install system packages before copying application code +- Copy requirements.txt before other files + +### 3.5 Base Image Modernization + +**Current:** `itisfoundation/jupyter-math:2.0.9` + +**Considerations:** +- Verify Ubuntu version (appears to be 20.04) +- Check for newer jupyter-math versions +- Consider updating to Ubuntu 22.04 (Jammy) for: + - Newer system packages + - Better security patches + - Modern library versions + +**Action Items:** +1. Document jupyter-math base image contents +2. Check for jupyter-math updates +3. Test compatibility with newer base versions +4. Evaluate custom base image if jupyter-math is outdated + +--- + +## 4. Implementation Roadmap + +### Phase 1: Documentation & Assessment (Week 1-2) + +**Tasks:** +1. ✅ Document current versions (this document) +2. ⬜ Build current image and extract all version information +3. ⬜ Create version inventory spreadsheet +4. ⬜ Research latest stable versions for all components +5. ⬜ Identify breaking changes between versions +6. ⬜ Document deprecated features + +**Deliverables:** +- Complete version inventory +- Research report on updates +- Risk assessment document + +### Phase 2: Dockerfile Refactoring (Week 3-4) + +**Tasks:** +1. ⬜ Create new multi-stage Dockerfile +2. ⬜ Implement build optimization techniques +3. ⬜ Add version pinning for all components +4. ⬜ Add inline documentation/comments +5. ⬜ Create .dockerignore file +6. ⬜ Set up BuildKit configuration + +**Deliverables:** +- Optimized Dockerfile +- Build time comparison report +- Image size comparison report + +### Phase 3: Version Updates - Conservative (Week 5-6) + +**Focus:** Update components with minimal breaking change risk + +**Tasks:** +1. ⬜ Update Python packages (minor versions) +2. ⬜ Update ANTs to latest 2.5.x +3. ⬜ Update c3d if newer version exists +4. ⬜ Pin Synb0-DISCO to specific commit +5. ⬜ Pin PyTorch to specific version +6. ⬜ Update MRtrix3 if patches available + +**Deliverables:** +- Updated Dockerfile (conservative) +- Test results for each component +- Rollback plan + +### Phase 4: Version Updates - Major (Week 7-10) + +**Focus:** Major version updates requiring extensive testing + +**Tasks:** +1. ⬜ Test FreeSurfer 7.x compatibility +2. ⬜ Update FSL to latest version +3. ⬜ Evaluate Spinal Cord Toolbox re-enablement +4. ⬜ Update base image if applicable +5. ⬜ Run full integration tests +6. ⬜ Benchmark performance changes + +**Deliverables:** +- Updated Dockerfile (with major updates) +- Comprehensive test suite results +- Performance comparison report +- Migration guide for users + +### Phase 5: Validation & Documentation (Week 11-12) + +**Tasks:** +1. ⬜ Run validation pipelines (Fariba_full_pipeline) +2. ⬜ Test all Jupyter notebooks +3. ⬜ Update README.md with new versions +4. ⬜ Create CHANGELOG.md +5. ⬜ Update docker-compose.yml +6. ⬜ Create upgrade guide for users +7. ⬜ Document known issues + +**Deliverables:** +- Validated service image +- Updated documentation +- Release notes +- User upgrade guide + +--- + +## 5. Testing Strategy + +### 5.1 Component-Level Testing + +For each updated component: +- ✅ Installation succeeds +- ✅ Version command works +- ✅ Basic functionality test +- ✅ Integration with other components + +### 5.2 Integration Testing + +**Test Cases:** +1. MRtrix3 pipeline with ANTs preprocessing +2. FreeSurfer recon-all workflow +3. FSL analysis pipeline +4. Synb0-DISCO distortion correction +5. Python package imports and basic operations +6. Jupyter notebook execution + +### 5.3 Performance Testing + +**Metrics:** +- Docker build time +- Image size (uncompressed/compressed) +- Container startup time +- Memory usage during typical workflows +- CPU utilization + +### 5.4 Regression Testing + +**Use existing validation:** +- Run existing validation scripts +- Compare outputs with previous version +- Ensure no functionality loss + +--- + +## 6. Risk Assessment & Mitigation + +### High Risk Updates + +| Component | Risk | Impact | Mitigation | +|-----------|------|--------|------------| +| FreeSurfer 6→7 | High | High | Parallel testing; keep v6 fallback | +| FSL major update | Medium | High | Pin specific version; extensive testing | +| Base image update | Medium | High | Test in isolated environment first | +| PyTorch update | Low | Medium | Use CPU-only; test inference | + +### Rollback Strategy + +1. **Git branching:** + - Create `modernization` branch + - Keep `main` stable + - Tag releases properly + +2. **Docker image tags:** + - Tag current version: `1.2.1-legacy` + - Tag new version: `1.3.0` + - Keep both available during transition + +3. **Feature flags:** + - Allow users to specify version preference + - Provide migration period + +--- + +## 7. Monitoring & Metrics + +### Build Metrics + +**Baseline (Current):** +- Build time: [TO BE MEASURED] +- Image size: [TO BE MEASURED] +- Number of layers: [TO BE MEASURED] + +**Target (Optimized):** +- Build time: -30% +- Image size: -20% +- Number of layers: <50 + +### Success Criteria + +1. ✅ All components updated to documented versions +2. ✅ Build time reduced by at least 20% +3. ✅ Image size reduced by at least 15% +4. ✅ All validation tests pass +5. ✅ No regression in functionality +6. ✅ Documentation complete and accurate + +--- + +## 8. Maintenance Plan + +### Version Tracking + +**Create `VERSION_MANIFEST.txt` in image:** +```txt +# Medical Image Processing Service Version Manifest +# Generated: [BUILD_DATE] + +SERVICE_VERSION=1.3.0 +BASE_IMAGE=itisfoundation/jupyter-math:2.0.9 + +# Medical Imaging Software +MRTRIX3_VERSION=3.0.4 +FREESURFER_VERSION=7.4.0 +FSL_VERSION=6.0.7.3 +ANTS_VERSION=2.5.0 +C3D_VERSION=1.0.0 +SYNB0_DISCO_VERSION=[commit_hash] + +# Python Packages +PYTHON_VERSION=[version] +PYTORCH_VERSION=[version] +NIBABEL_VERSION=[version] +PYVISTA_VERSION=[version] +FSLEYES_VERSION=[version] +CC3D_VERSION=[version] +``` + +### Update Schedule + +- **Monthly:** Check for security updates +- **Quarterly:** Review for minor version updates +- **Semi-annually:** Evaluate major version updates +- **Annually:** Full modernization review + +### Documentation Requirements + +For each update: +1. Update VERSION_MANIFEST.txt +2. Update CHANGELOG.md +3. Update README.md version references +4. Update docker-compose.yml labels +5. Tag git commit with version +6. Create GitHub release with notes + +--- + +## 9. Quick Reference Commands + +### Version Checking Commands + +```bash +# Build and check current versions +docker build -t medimproc:test . +docker run --rm medimproc:test /bin/bash -c " + echo 'MRtrix3:' && mrinfo --version + echo 'FreeSurfer:' && cat \${FREESURFER_HOME}/build-stamp.txt + echo 'FSL:' && cat \${FSLDIR}/etc/fslversion + echo 'ANTs:' && antsRegistration --version + echo 'Python packages:' && pip list +" + +# Check image size +docker images medimproc:test --format "{{.Size}}" + +# Check layer count +docker history medimproc:test --no-trunc | wc -l + +# Analyze image +docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ + wagoodman/dive:latest medimproc:test +``` + +### Build with BuildKit + +```bash +# Enable BuildKit +export DOCKER_BUILDKIT=1 +export COMPOSE_DOCKER_CLI_BUILD=1 + +# Build with cache +docker build --build-arg BUILDKIT_INLINE_CACHE=1 -t medimproc:test . + +# Build with progress +docker build --progress=plain -t medimproc:test . +``` + +--- + +## 10. Resources & References + +### Official Documentation +- [MRtrix3](https://mrtrix.readthedocs.io/) +- [FreeSurfer](https://surfer.nmr.mgh.harvard.edu/fswiki) +- [FSL](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki) +- [ANTs](http://stnava.github.io/ANTs/) +- [Synb0-DISCO](https://github.com/MASILab/Synb0-DISCO) +- [Spinal Cord Toolbox](https://spinalcordtoolbox.com/overview/introduction.html) + +### Docker Best Practices +- [Dockerfile best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) +- [Multi-stage builds](https://docs.docker.com/build/building/multi-stage/) +- [BuildKit](https://docs.docker.com/build/buildkit/) + +### Optimization Tools +- [dive - Docker image analyzer](https://github.com/wagoodman/dive) +- [hadolint - Dockerfile linter](https://github.com/hadolint/hadolint) +- [docker-slim](https://github.com/docker-slim/docker-slim) + +--- + +## Appendix A: Sample Optimized Dockerfile Structure + +```dockerfile +# syntax=docker/dockerfile:1.4 + +# ============================================================================== +# Stage 1: Base Builder +# ============================================================================== +FROM buildpack-deps:jammy AS base-builder +ENV DEBIAN_FRONTEND=noninteractive +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + ca-certificates \ + curl \ + git \ + wget && \ + rm -rf /var/lib/apt/lists/* + +# ============================================================================== +# Stage 2: MRtrix3 Builder +# ============================================================================== +FROM base-builder AS mrtrix3-builder +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + g++ \ + libeigen3-dev \ + libfftw3-dev \ + libgl1-mesa-dev \ + libpng-dev \ + libqt5opengl5-dev \ + libqt5svg5-dev \ + libtiff5-dev \ + python3 \ + qt5-qmake \ + qtbase5-dev \ + zlib1g-dev && \ + rm -rf /var/lib/apt/lists/* + +WORKDIR /opt/mrtrix3 +RUN --mount=type=cache,target=/root/.cache \ + git clone --depth 1 --branch 3.0.4 https://github.com/MRtrix3/mrtrix3.git . && \ + ./configure && \ + ./build -persistent -nopaginate && \ + rm -rf .git tmp + +# ============================================================================== +# Stage 3: Additional stages for other components... +# ============================================================================== + +# ============================================================================== +# Final Production Stage +# ============================================================================== +FROM itisfoundation/jupyter-math:2.0.9 AS production +LABEL maintainer="ordonez" + +# Copy only runtime components from builders +COPY --from=mrtrix3-builder /opt/mrtrix3 ${HOME}/mrtrix3 + +# ... continue with other components ... +``` + +--- + +## Appendix B: Migration Checklist + +- [ ] Back up current working image +- [ ] Create git branch for modernization +- [ ] Document current versions +- [ ] Research updates for all components +- [ ] Create optimized Dockerfile +- [ ] Test build process +- [ ] Validate image size reduction +- [ ] Run component tests +- [ ] Run integration tests +- [ ] Update documentation +- [ ] Create release notes +- [ ] Tag release +- [ ] Deploy to test environment +- [ ] User acceptance testing +- [ ] Production deployment + +--- + +**Document Version:** 1.0 +**Last Updated:** December 1, 2025 +**Next Review:** After Phase 1 completion diff --git a/README.md b/README.md index 0515237..c6707d2 100755 --- a/README.md +++ b/README.md @@ -14,6 +14,36 @@ Either single commands can be run, or full .sh files. ____ +## Service Modernization + +**Current Version:** 1.2.1 + +A comprehensive modernization plan is available in [`MODERNIZATION_PLAN.md`](./MODERNIZATION_PLAN.md). This plan covers: + +- **Current Software Versions:** Detailed inventory of all installed medical imaging software and their versions +- **Version Updates:** Investigation of newer versions and update strategies for all components +- **Dockerfile Optimization:** Multi-stage build implementation and layer optimization techniques +- **Image Size Reduction:** Strategies to reduce the Docker image size by ~15-20% +- **Build Time Optimization:** Techniques to reduce build time by ~20-30% +- **Implementation Roadmap:** Phased approach for modernization over 12 weeks +- **Testing & Validation:** Comprehensive testing strategy to ensure no regression + +### Quick Version Reference + +| Component | Current Version | Location in Dockerfile | +|-----------|----------------|------------------------| +| Base Image | jupyter-math:2.0.9 | Line 3 | +| MRtrix3 | 3.0.4 | Lines 23-27 | +| FreeSurfer | 6.0.0 | Lines 41-49 | +| FSL | Not pinned | Lines 57-63 | +| ANTs | 2.4.4 | Lines 105-107 | +| Synb0-DISCO | master (unpinned) | Lines 75-103 | +| Spinal Cord Toolbox | 4.2.1 (disabled) | Lines 112-115 | + +For detailed version information, update strategies, and optimization techniques, please refer to the [Modernization Plan](./MODERNIZATION_PLAN.md). + +____ + ## Information for developers of this **o2S2PARC** service Building the docker image: diff --git a/UPDATING_FREESURFER.md b/UPDATING_FREESURFER.md new file mode 100644 index 0000000..588be83 --- /dev/null +++ b/UPDATING_FREESURFER.md @@ -0,0 +1,21 @@ +## Plan: Update FreeSurfer from v6.0 to v7 or v8 + +Based on the research, upgrading FreeSurfer will require installation method changes. The current tar.gz extraction approach will still work for v7, but using Debian packages (`.deb`) is now preferred and handles dependencies automatically. FreeSurfer v8 strongly recommends package installation and requires significantly more RAM (32GB vs 16GB). + +### Steps + +1. **Choose target version** - FreeSurfer v7.4.1 recommended over v8 due to lower RAM requirements (16GB vs 32GB) unless container deployment can guarantee 32GB+ memory. + +2. **Update FreeSurfer installation block in [`Dockerfile`](Dockerfile:47-56)** - Replace `wget` tar.gz extraction with `.deb` package installation using `apt-get install`, update FTP URL to HTTPS download URL, and verify dependencies are automatically handled. + +3. **Update license handling** - For v7, keep copying freesurfer_license.txt to `${FREESURFER_HOME}/license.txt` (no change needed); for v8, optionally switch to `FS_LICENSE` environment variable pointing to `/root/license.txt` for cleaner multi-version support. + +4. **Verify environment variables remain unchanged** - All existing `FREESURFER_HOME`, `FSFAST_HOME`, `MINC_BIN_DIR`, `MNI_DIR`, `PERL5LIB`, and `PATH` settings are compatible across v6, v7, and v8. + +### Further Considerations + +1. **Version selection** - Choose v7.4.1 for compatibility with current 16GB RAM containers, or v8.1.0 if 32GB+ RAM is available and faster processing (1.8-2.5hrs vs 10+hrs for recon-all) is desired? + +2. **Fallback for v8 memory constraints** - If upgrading to v8 with limited RAM, add `ENV FS_V8_XOPTS=0` to revert to older, less memory-intensive recon-all behavior? + +3. **Testing downstream dependencies** - Verify that existing pipeline scripts in Fariba_full_pipeline and validation workflows are compatible with the chosen FreeSurfer version after upgrade? \ No newline at end of file From ae20946f8808d3b87c1d971118b33f7769e70957 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Tue, 2 Dec 2025 15:08:33 +0100 Subject: [PATCH 02/19] update freesurfer using latest Docker Hub image --- Dockerfile | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/Dockerfile b/Dockerfile index 2f4f6c8..98dc376 100755 --- a/Dockerfile +++ b/Dockerfile @@ -1,5 +1,7 @@ -## Refactor as normal (ie no multistage) dockerfile -## also have a smaller base image +## Multi-stage build: use FreeSurfer official Docker image to avoid slow server downloads +FROM freesurfer/freesurfer:7.4.1@sha256:10b6468cbd9fcd2db3708f4651d59ad75d4da849a2c5d8bb6dba217f08b8c46b as freesurfer-source + +## Main build stage FROM itisfoundation/jupyter-math:2.0.9 as main LABEL maintainer="ordonez" USER root @@ -40,19 +42,22 @@ ENV ANTSPATH="$HOME/ants/bin" \ PATH="$HOME/mrtrix3/bin:$HOME/ants/bin:$HOME/art/bin:$PATH" ############################################################ -## Freesurfer +## Freesurfer v7.4.1 (copied from official Docker image to avoid slow download) WORKDIR ${HOME} +# Install FreeSurfer runtime dependencies RUN apt-get update && apt-get install -y tcsh bc libgomp1 perl-modules \ && rm -rf /var/lib/apt/lists/* -RUN wget -N -qO- ftp://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/6.0.0/freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.0.tar.gz | tar -xzv -C ${HOME} -# \ && rm -rf ${HOME}/freesurfer/subjects # we actually need subjects/fsaverage for recon-all -ENV FREESURFER_HOME ${HOME}/freesurfer -COPY freesurfer_license.txt ${FREESURFER_HOME}/license.txt -ENV FSFAST_HOME==$FREESURFER_HOME/fsfast \ - MINC_BIN_DIR=$FREESURFER_HOME/mni/bin \ - MNI_DIR=$FREESURFER_HOME/mni \ - PERL5LIB=$FREESURFER_HOME/mni/share/perl5 +# Copy FreeSurfer installation from official Docker image +COPY --from=freesurfer-source /usr/local/freesurfer /usr/local/freesurfer +# Set up FreeSurfer environment variables +ENV FREESURFER_HOME=/usr/local/freesurfer/7.4.1 \ + FSFAST_HOME=/usr/local/freesurfer/7.4.1/fsfast \ + MINC_BIN_DIR=/usr/local/freesurfer/7.4.1/mni/bin \ + MNI_DIR=/usr/local/freesurfer/7.4.1/mni \ + PERL5LIB=/usr/local/freesurfer/7.4.1/mni/share/perl5 ENV PATH=$FREESURFER_HOME/bin:$MINC_BIN_DIR:$PATH +# Copy license file to FreeSurfer home (v7 method) +COPY freesurfer_license.txt $FREESURFER_HOME/license.txt ############################################################ ## FSL From f23fc60af7fce9b8235404aa5dbb75f26c962404 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Tue, 2 Dec 2025 15:21:13 +0100 Subject: [PATCH 03/19] pin requirements --- requirements.in | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/requirements.in b/requirements.in index b62f4d1..ca3903b 100755 --- a/requirements.in +++ b/requirements.in @@ -1,7 +1,9 @@ -nibabel -pyvista +# Medical image processing and visualization packages +# Pinned versions for reproducibility (checked December 2025) -PyOpenGL -PyOpenGL_accelerate -fsleyes -connected-components-3d \ No newline at end of file +nibabel==5.3.2 # 23 Oct 2024 +pyvista==0.46.4 # 30 Oct 2025 +PyOpenGL==3.1.10 # 18 Aug 2025 +PyOpenGL_accelerate==3.1.10 # 18 Aug 2025 +fsleyes==1.16.3 # 21 Nov 2025 +connected-components-3d==3.26.1 # 3 Nov 2025 From 3d7628e8b590ee0f46de69e8e053c9a3bca77b55 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Tue, 2 Dec 2025 15:22:01 +0100 Subject: [PATCH 04/19] add dockerignore --- .dockerignore | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 .dockerignore diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..12ae213 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,50 @@ +# Git files +.git +.gitignore +.gitattributes + +# Documentation +*.md +!README.md +UPDATING_FREESURFER.md +MODERNIZATION_PLAN.md + +# Docker files (except those needed for build) +docker-compose*.yml +MultiStage_Dockerfile + +# Validation and test data +validation/ +Fariba_full_pipeline/TVB1/ + +# Python cache +**/__pycache__ +**/*.pyc +**/*.pyo +**/*.pyd +.Python +*.so +*.egg +*.egg-info +dist/ +build/ + +# IDE and editor files +.vscode/ +.idea/ +*.swp +*.swo +*~ +.DS_Store + +# Temporary files +*.tmp +*.log +.cache/ + +# Jupyter notebook checkpoints +.ipynb_checkpoints/ + +# Only include specific notebooks needed in the image +validation/workspace/*.ipynb +!README.ipynb From b3c97dc96016ba4f02f525eb6906def1995a054a Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 10:17:25 +0100 Subject: [PATCH 05/19] [FIX] build runs again through FSLeyes & wxpython --- Dockerfile | 15 ++++++++++----- README.md | 9 +++++++++ 2 files changed, 19 insertions(+), 5 deletions(-) diff --git a/Dockerfile b/Dockerfile index 98dc376..fefc394 100755 --- a/Dockerfile +++ b/Dockerfile @@ -19,7 +19,10 @@ RUN apt-get -qq update \ libqt5opengl5-dev \ libqt5svg5-dev \ libtiff5-dev \ - qt5-default \ + qtbase5-dev \ + qtchooser \ + qt5-qmake \ + qtbase5-dev-tools \ zlib1g-dev \ && rm -rf /var/lib/apt/lists/* @@ -124,17 +127,19 @@ ENV PATH=$PATH:$HOME/c3d-1.0.0-Linux-x86_64/bin/ ############################################################ ## python packages in requirements.in -## before pip install fsleyes, we need to install wxPython: +## fsleyes requires wxPython, which has pre-built wheels available WORKDIR ${HOME} -RUN .venv/bin/pip --no-cache install -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-20.04 wxpython &&\ - .venv/bin/pip install attrdict COPY --chown=$NB_UID:$NB_GID requirements.in ${NOTEBOOK_BASE_DIR}/requirements.in RUN .venv/bin/pip --no-cache install pip-tools &&\ ## rename the previously existing "requirements.txt" from the jupyter-math service (we want to keep it for user reference) mv ${NOTEBOOK_BASE_DIR}/requirements.txt ${NOTEBOOK_BASE_DIR}/requirements_base_math.txt &&\ + ## Run pip-compile with find-links and only-binary to ensure wxpython wheel is used + PIP_FIND_LINKS=https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-20.04 \ + PIP_ONLY_BINARY=wxpython \ .venv/bin/pip-compile --build-isolation --output-file ${NOTEBOOK_BASE_DIR}/requirements.txt ${NOTEBOOK_BASE_DIR}/requirements.in &&\ - .venv/bin/pip --no-cache install -r ${NOTEBOOK_BASE_DIR}/requirements.txt && \ + ## Install all packages, ensuring wxpython comes from the wheel + .venv/bin/pip --no-cache install --only-binary=wxpython --find-links https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-20.04 -r ${NOTEBOOK_BASE_DIR}/requirements.txt && \ rm ${NOTEBOOK_BASE_DIR}/requirements.in && \ echo "Your environment contains these python packages:" && \ .venv/bin/pip list diff --git a/README.md b/README.md index c6707d2..776460f 100755 --- a/README.md +++ b/README.md @@ -1,5 +1,14 @@ # JupyterLab for Medical Image Processing (MedImProc) + + This is the source code of the JupyterLab Medical Image Processing OSPARC service. It is mostly centered on MRI data, and contains the following packages: "[JupyterLab](https://jupyter.org/) with a variety of Medical Image Processing packages pre-installed, mostly centered on MRI data: - [FSL](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki), a comprehensive library of analysis tools for FMRI, MRI and DTI brain imaging data. From 3ac5809f147554ab27c89da5db6ce23b24b8928e Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 10:23:31 +0100 Subject: [PATCH 06/19] cleaning up --- MODERNIZATION_PLAN.md | 751 ----------------------------------------- README.md | 34 +- UPDATING_FREESURFER.md | 21 -- 3 files changed, 2 insertions(+), 804 deletions(-) delete mode 100644 MODERNIZATION_PLAN.md delete mode 100644 UPDATING_FREESURFER.md diff --git a/MODERNIZATION_PLAN.md b/MODERNIZATION_PLAN.md deleted file mode 100644 index e1c03a4..0000000 --- a/MODERNIZATION_PLAN.md +++ /dev/null @@ -1,751 +0,0 @@ -# Service Modernization Plan: JupyterLab Medical Image Processing - -**Date:** December 1, 2025 -**Service Version:** 1.2.1 -**Maintainer:** ordonez - ---- - -## Executive Summary - -This document outlines a comprehensive plan to modernize the JupyterLab Medical Image Processing (MedImProc) service. The modernization focuses on documenting current software versions, updating to newer releases, optimizing build times, and reducing image size through improved Dockerfile practices. - ---- - -## 1. Current Status Documentation - -### 1.1 Base Image -- **Base:** `itisfoundation/jupyter-math:2.0.9` -- **Status:** Inherited from upstream, requires investigation -- **How to verify:** Check Docker Hub or source repository for jupyter-math - -### 1.2 Medical Image Processing Software Versions - -#### MRtrix3 -- **Current Version:** 3.0.4 (Git checkout) -- **Installation Method:** Compiled from source (GitHub) -- **Repository:** https://github.com/MRtrix3/mrtrix3.git -- **How to check version:** - ```bash - docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 /bin/bash -c "mrinfo --version" - ``` -- **Location in Dockerfile:** Lines 23-27 - -#### FreeSurfer -- **Current Version:** 6.0.0 -- **Installation Method:** Pre-compiled binary (FTP download) -- **Download URL:** ftp://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/6.0.0/freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.0.tar.gz -- **How to check version:** - ```bash - docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 /bin/bash -c "source ${FREESURFER_HOME}/SetUpFreeSurfer.sh && freesurfer --version" - ``` -- **Location in Dockerfile:** Lines 41-49 -- **Note:** Requires license file (freesurfer_license.txt) - -#### FSL -- **Current Version:** Unknown (downloaded via installer.py) -- **Installation Method:** Python installer script -- **Download URL:** https://fsl.fmrib.ox.ac.uk/fsldownloads/fslinstaller.py -- **How to check version:** - ```bash - docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 /bin/bash -c "cat ${FSLDIR}/etc/fslversion" - ``` -- **Location in Dockerfile:** Lines 57-63 -- **Issue:** Version not explicitly pinned, downloads latest available at build time - -#### ANTs (for Synb0-DISCO) -- **Current Version:** 2.4.4 -- **Installation Method:** Pre-compiled binary (GitHub releases) -- **Download URL:** https://github.com/ANTsX/ANTs/releases/download/v2.4.4/ants-2.4.4-ubuntu-20.04-X64-gcc.zip -- **How to check version:** - ```bash - docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 antsRegistration --version - ``` -- **Location in Dockerfile:** Lines 105-107 - -#### c3d (Convert3D) -- **Current Version:** 1.0.0 -- **Installation Method:** Pre-compiled binary (SourceForge) -- **Download URL:** https://sourceforge.net/projects/c3d/files/c3d/1.0.0/c3d-1.0.0-Linux-x86_64.tar.gz -- **How to check version:** - ```bash - docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 c3d -version - ``` -- **Location in Dockerfile:** Line 109 - -#### Synb0-DISCO -- **Current Version:** master branch (unpinned) -- **Installation Method:** Git clone from GitHub -- **Repository:** https://github.com/MASILab/Synb0-DISCO -- **How to check version:** - ```bash - docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 /bin/bash -c "cd ${HOME}/synb0-disco && git log -1 --format='%H %ci'" - ``` -- **Location in Dockerfile:** Lines 75-103 -- **Issue:** Not pinned to specific version, uses master branch -- **Dependencies:** PyTorch (CPU version), installed via pip - -#### Spinal Cord Toolbox (SCT) -- **Current Version:** 4.2.1 -- **Status:** TEMPORARILY REMOVED (commented out in Dockerfile) -- **Reason:** GitHub building space issues (see lines 112-115) -- **Installation Method:** When enabled, downloads from GitHub releases -- **Download URL:** https://github.com/neuropoly/spinalcordtoolbox/archive/4.2.1.tar.gz -- **Note:** Adds ~1.5GB to image - -#### ART ACPCdetect (for MRtrix3) -- **Current Version:** 2.0 (minified) -- **Installation Method:** Pre-compiled binary (OSF download) -- **Download URL:** https://osf.io/73h5s/download -- **Location in Dockerfile:** Lines 29-31 - -#### ANTs for MRtrix3 -- **Current Version:** 2.3.4-2 (minified) -- **Installation Method:** Pre-compiled binary (OSF download) -- **Download URL:** https://osf.io/yswa4/download -- **Location in Dockerfile:** Lines 33-35 -- **Note:** Different ANTs version than the one used for Synb0-DISCO - -### 1.3 Python Packages -Defined in `requirements.in`: -- **nibabel** - Version: Unknown (not pinned) -- **pyvista** - Version: Unknown (not pinned) -- **PyOpenGL** - Version: Unknown (not pinned) -- **PyOpenGL_accelerate** - Version: Unknown (not pinned) -- **fsleyes** - Version: Unknown (not pinned) -- **connected-components-3d** - Version: Unknown (not pinned) -- **wxpython** - Installed separately with Ubuntu 20.04 wheels -- **attrdict** - Installed separately -- **PyTorch, torchvision, torchaudio** - CPU version (not in requirements.in) - -**How to check versions:** -```bash -docker run simcore/services/dynamic/jupyter-medimproc:1.2.1 /bin/bash -c "source .venv/bin/activate && pip list" -``` - -**Generated file:** `requirements.txt` is compiled from `requirements.in` using pip-tools during build - -### 1.4 System Dependencies -- Base OS: Ubuntu 20.04 (inherited from jupyter-math base image) -- Key libraries: - - Qt5 (for MRtrix3 GUI components) - - FFTW3 (for MRtrix3) - - Eigen3 (for MRtrix3) - - OpenGL/Mesa libraries - - Various system utilities (curl, git, jq, etc.) - ---- - -## 2. Version Investigation Plan - -### 2.1 Priority 1: Critical Software (High Impact) - -#### FSL -- **Action:** Determine current installed version -- **Method:** - 1. Build current image - 2. Run: `cat ${FSLDIR}/etc/fslversion` - 3. Check FSL website for latest version: https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FslInstallation -- **Latest Known:** FSL 6.0.7.x (as of 2024) -- **Update Strategy:** Pin to specific version by modifying installer script or using direct download URL - -#### MRtrix3 -- **Current:** 3.0.4 -- **Latest Check:** https://github.com/MRtrix3/mrtrix3/releases -- **Latest Known:** 3.0.4 is latest stable release (as of check date) -- **Action:** Verify if any newer patches or releases are available - -#### PyTorch -- **Current:** CPU version, unpinned -- **Action:** Pin to specific version -- **Latest Check:** https://pytorch.org/get-started/previous-versions/ -- **Recommendation:** Use latest stable CPU-only version compatible with Python version in base image - -### 2.2 Priority 2: Moderate Updates - -#### FreeSurfer -- **Current:** 6.0.0 (from 2016) -- **Latest Check:** https://surfer.nmr.mgh.harvard.edu/fswiki/DownloadAndInstall -- **Latest Known:** FreeSurfer 7.4.x (as of 2024) -- **Consideration:** Major version update may require significant testing -- **Action:** Test FreeSurfer 7.x compatibility with existing pipelines - -#### ANTs -- **Current:** 2.4.4 (for Synb0), 2.3.4-2 (for MRtrix3) -- **Latest Check:** https://github.com/ANTsX/ANTs/releases -- **Latest Known:** 2.5.x -- **Issue:** Two different versions installed for different purposes -- **Action:** Consolidate to single ANTs installation if possible - -#### Synb0-DISCO -- **Current:** master branch (unpinned) -- **Action:** Pin to specific commit hash or tag -- **Latest Check:** https://github.com/MASILab/Synb0-DISCO/releases -- **Recommendation:** Use latest tagged release or pin commit hash - -### 2.3 Priority 3: Python Packages - -#### All Python packages in requirements.in -- **Action:** Create `requirements-versions.txt` with pinned versions -- **Method:** - 1. Build current image - 2. Export: `pip freeze > requirements-pinned.txt` - 3. Research latest versions on PyPI - 4. Test compatibility before updating - -#### Spinal Cord Toolbox -- **Current:** 4.2.1 (DISABLED) -- **Latest Check:** https://github.com/neuropoly/spinalcordtoolbox/releases -- **Latest Known:** 6.x -- **Action:** Evaluate if re-enabling is needed; if so, update to latest version - ---- - -## 3. Dockerfile Optimization Strategy - -### 3.1 Multi-Stage Build Implementation - -**Current Status:** -- Single-stage Dockerfile (current production) -- Incomplete multi-stage Dockerfile exists (MultiStage_Dockerfile) -- MultiStage_Dockerfile appears to be work-in-progress - -**Proposed Multi-Stage Architecture:** - -```dockerfile -# Stage 1: Base builder image with common build tools -FROM buildpack-deps:jammy AS base-builder - -# Stage 2: MRtrix3 builder -FROM base-builder AS mrtrix3-builder -# Compile MRtrix3 with dependencies - -# Stage 3: FreeSurfer downloader -FROM base-builder AS freesurfer-installer -# Download and extract FreeSurfer - -# Stage 4: FSL installer -FROM base-builder AS fsl-installer -# Install FSL - -# Stage 5: Synb0-DISCO setup -FROM base-builder AS synb0-installer -# Clone and setup Synb0-DISCO - -# Stage 6: Python environment builder -FROM base AS python-builder -# Install Python packages in venv - -# Stage 7: Final production image -FROM itisfoundation/jupyter-math:2.0.9 AS production -# Copy only necessary artifacts from previous stages -``` - -**Benefits:** -- Parallel builds for independent components -- Smaller final image (no build tools) -- Better layer caching -- Easier to maintain and update individual components - -### 3.2 Layer Optimization Strategies - -#### A. Combine RUN Commands -**Current Problem:** Multiple separate RUN commands create unnecessary layers - -**Solution:** -```dockerfile -# Before (multiple layers) -RUN apt-get update -RUN apt-get install -y package1 -RUN apt-get install -y package2 - -# After (single layer) -RUN apt-get update && \ - apt-get install -y --no-install-recommends \ - package1 \ - package2 && \ - rm -rf /var/lib/apt/lists/* -``` - -#### B. Minimize Package Installation -- Use `--no-install-recommends` flag -- Remove package manager cache in same layer -- Audit installed packages for necessity - -#### C. Optimize File Operations -- Download and extract in single RUN command -- Clean up temporary files in same layer -- Remove .git directories after clone - -### 3.3 Image Size Reduction Tactics - -#### Current Issues: -1. **FreeSurfer subjects directory:** Kept for fsaverage (needed for recon-all) -2. **Multiple ANTs installations:** 2.3.4-2 and 2.4.4 both present -3. **Build artifacts:** Some tools compiled from source leave build files -4. **Uncompressed downloads:** Large tarballs downloaded and not cleaned up -5. **Git history:** Full git clones with history - -#### Proposed Solutions: - -**1. FreeSurfer Optimization:** -```dockerfile -# Keep only essential subjects (fsaverage) and remove others -RUN rm -rf ${FREESURFER_HOME}/subjects/cvs_avg35* \ - ${FREESURFER_HOME}/subjects/bert \ - ${FREESURFER_HOME}/trctrain -``` - -**2. Consolidate ANTs:** -```dockerfile -# Use single ANTs version for all purposes -# Test compatibility first -``` - -**3. Shallow Git Clones:** -```dockerfile -# Before -RUN git clone https://github.com/MRtrix3/mrtrix3.git - -# After -RUN git clone --depth 1 --branch 3.0.4 https://github.com/MRtrix3/mrtrix3.git && \ - cd mrtrix3 && rm -rf .git -``` - -**4. Cleanup Build Dependencies:** -```dockerfile -RUN apt-get update && \ - apt-get install -y --no-install-recommends \ - build-essential \ - git \ - # ... build dependencies ... && \ - # ... build software ... && \ - apt-get remove -y build-essential git && \ - apt-get autoremove -y && \ - rm -rf /var/lib/apt/lists/* -``` - -### 3.4 Build Time Optimization - -#### A. Use BuildKit -```dockerfile -# syntax=docker/dockerfile:1.4 -# Enable BuildKit features -``` - -**Features to leverage:** -- `RUN --mount=type=cache` for package manager caches -- `RUN --mount=type=bind` for temporary file access -- Parallel stage execution - -#### B. Optimize Download Steps -```dockerfile -# Use --link flag for COPY (BuildKit) -COPY --link --from=builder /app /app - -# Cache pip downloads -RUN --mount=type=cache,target=/root/.cache/pip \ - pip install -r requirements.txt -``` - -#### C. Order Dockerfile for Cache Efficiency -- Put frequently changing commands last -- Install system packages before copying application code -- Copy requirements.txt before other files - -### 3.5 Base Image Modernization - -**Current:** `itisfoundation/jupyter-math:2.0.9` - -**Considerations:** -- Verify Ubuntu version (appears to be 20.04) -- Check for newer jupyter-math versions -- Consider updating to Ubuntu 22.04 (Jammy) for: - - Newer system packages - - Better security patches - - Modern library versions - -**Action Items:** -1. Document jupyter-math base image contents -2. Check for jupyter-math updates -3. Test compatibility with newer base versions -4. Evaluate custom base image if jupyter-math is outdated - ---- - -## 4. Implementation Roadmap - -### Phase 1: Documentation & Assessment (Week 1-2) - -**Tasks:** -1. ✅ Document current versions (this document) -2. ⬜ Build current image and extract all version information -3. ⬜ Create version inventory spreadsheet -4. ⬜ Research latest stable versions for all components -5. ⬜ Identify breaking changes between versions -6. ⬜ Document deprecated features - -**Deliverables:** -- Complete version inventory -- Research report on updates -- Risk assessment document - -### Phase 2: Dockerfile Refactoring (Week 3-4) - -**Tasks:** -1. ⬜ Create new multi-stage Dockerfile -2. ⬜ Implement build optimization techniques -3. ⬜ Add version pinning for all components -4. ⬜ Add inline documentation/comments -5. ⬜ Create .dockerignore file -6. ⬜ Set up BuildKit configuration - -**Deliverables:** -- Optimized Dockerfile -- Build time comparison report -- Image size comparison report - -### Phase 3: Version Updates - Conservative (Week 5-6) - -**Focus:** Update components with minimal breaking change risk - -**Tasks:** -1. ⬜ Update Python packages (minor versions) -2. ⬜ Update ANTs to latest 2.5.x -3. ⬜ Update c3d if newer version exists -4. ⬜ Pin Synb0-DISCO to specific commit -5. ⬜ Pin PyTorch to specific version -6. ⬜ Update MRtrix3 if patches available - -**Deliverables:** -- Updated Dockerfile (conservative) -- Test results for each component -- Rollback plan - -### Phase 4: Version Updates - Major (Week 7-10) - -**Focus:** Major version updates requiring extensive testing - -**Tasks:** -1. ⬜ Test FreeSurfer 7.x compatibility -2. ⬜ Update FSL to latest version -3. ⬜ Evaluate Spinal Cord Toolbox re-enablement -4. ⬜ Update base image if applicable -5. ⬜ Run full integration tests -6. ⬜ Benchmark performance changes - -**Deliverables:** -- Updated Dockerfile (with major updates) -- Comprehensive test suite results -- Performance comparison report -- Migration guide for users - -### Phase 5: Validation & Documentation (Week 11-12) - -**Tasks:** -1. ⬜ Run validation pipelines (Fariba_full_pipeline) -2. ⬜ Test all Jupyter notebooks -3. ⬜ Update README.md with new versions -4. ⬜ Create CHANGELOG.md -5. ⬜ Update docker-compose.yml -6. ⬜ Create upgrade guide for users -7. ⬜ Document known issues - -**Deliverables:** -- Validated service image -- Updated documentation -- Release notes -- User upgrade guide - ---- - -## 5. Testing Strategy - -### 5.1 Component-Level Testing - -For each updated component: -- ✅ Installation succeeds -- ✅ Version command works -- ✅ Basic functionality test -- ✅ Integration with other components - -### 5.2 Integration Testing - -**Test Cases:** -1. MRtrix3 pipeline with ANTs preprocessing -2. FreeSurfer recon-all workflow -3. FSL analysis pipeline -4. Synb0-DISCO distortion correction -5. Python package imports and basic operations -6. Jupyter notebook execution - -### 5.3 Performance Testing - -**Metrics:** -- Docker build time -- Image size (uncompressed/compressed) -- Container startup time -- Memory usage during typical workflows -- CPU utilization - -### 5.4 Regression Testing - -**Use existing validation:** -- Run existing validation scripts -- Compare outputs with previous version -- Ensure no functionality loss - ---- - -## 6. Risk Assessment & Mitigation - -### High Risk Updates - -| Component | Risk | Impact | Mitigation | -|-----------|------|--------|------------| -| FreeSurfer 6→7 | High | High | Parallel testing; keep v6 fallback | -| FSL major update | Medium | High | Pin specific version; extensive testing | -| Base image update | Medium | High | Test in isolated environment first | -| PyTorch update | Low | Medium | Use CPU-only; test inference | - -### Rollback Strategy - -1. **Git branching:** - - Create `modernization` branch - - Keep `main` stable - - Tag releases properly - -2. **Docker image tags:** - - Tag current version: `1.2.1-legacy` - - Tag new version: `1.3.0` - - Keep both available during transition - -3. **Feature flags:** - - Allow users to specify version preference - - Provide migration period - ---- - -## 7. Monitoring & Metrics - -### Build Metrics - -**Baseline (Current):** -- Build time: [TO BE MEASURED] -- Image size: [TO BE MEASURED] -- Number of layers: [TO BE MEASURED] - -**Target (Optimized):** -- Build time: -30% -- Image size: -20% -- Number of layers: <50 - -### Success Criteria - -1. ✅ All components updated to documented versions -2. ✅ Build time reduced by at least 20% -3. ✅ Image size reduced by at least 15% -4. ✅ All validation tests pass -5. ✅ No regression in functionality -6. ✅ Documentation complete and accurate - ---- - -## 8. Maintenance Plan - -### Version Tracking - -**Create `VERSION_MANIFEST.txt` in image:** -```txt -# Medical Image Processing Service Version Manifest -# Generated: [BUILD_DATE] - -SERVICE_VERSION=1.3.0 -BASE_IMAGE=itisfoundation/jupyter-math:2.0.9 - -# Medical Imaging Software -MRTRIX3_VERSION=3.0.4 -FREESURFER_VERSION=7.4.0 -FSL_VERSION=6.0.7.3 -ANTS_VERSION=2.5.0 -C3D_VERSION=1.0.0 -SYNB0_DISCO_VERSION=[commit_hash] - -# Python Packages -PYTHON_VERSION=[version] -PYTORCH_VERSION=[version] -NIBABEL_VERSION=[version] -PYVISTA_VERSION=[version] -FSLEYES_VERSION=[version] -CC3D_VERSION=[version] -``` - -### Update Schedule - -- **Monthly:** Check for security updates -- **Quarterly:** Review for minor version updates -- **Semi-annually:** Evaluate major version updates -- **Annually:** Full modernization review - -### Documentation Requirements - -For each update: -1. Update VERSION_MANIFEST.txt -2. Update CHANGELOG.md -3. Update README.md version references -4. Update docker-compose.yml labels -5. Tag git commit with version -6. Create GitHub release with notes - ---- - -## 9. Quick Reference Commands - -### Version Checking Commands - -```bash -# Build and check current versions -docker build -t medimproc:test . -docker run --rm medimproc:test /bin/bash -c " - echo 'MRtrix3:' && mrinfo --version - echo 'FreeSurfer:' && cat \${FREESURFER_HOME}/build-stamp.txt - echo 'FSL:' && cat \${FSLDIR}/etc/fslversion - echo 'ANTs:' && antsRegistration --version - echo 'Python packages:' && pip list -" - -# Check image size -docker images medimproc:test --format "{{.Size}}" - -# Check layer count -docker history medimproc:test --no-trunc | wc -l - -# Analyze image -docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ - wagoodman/dive:latest medimproc:test -``` - -### Build with BuildKit - -```bash -# Enable BuildKit -export DOCKER_BUILDKIT=1 -export COMPOSE_DOCKER_CLI_BUILD=1 - -# Build with cache -docker build --build-arg BUILDKIT_INLINE_CACHE=1 -t medimproc:test . - -# Build with progress -docker build --progress=plain -t medimproc:test . -``` - ---- - -## 10. Resources & References - -### Official Documentation -- [MRtrix3](https://mrtrix.readthedocs.io/) -- [FreeSurfer](https://surfer.nmr.mgh.harvard.edu/fswiki) -- [FSL](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki) -- [ANTs](http://stnava.github.io/ANTs/) -- [Synb0-DISCO](https://github.com/MASILab/Synb0-DISCO) -- [Spinal Cord Toolbox](https://spinalcordtoolbox.com/overview/introduction.html) - -### Docker Best Practices -- [Dockerfile best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) -- [Multi-stage builds](https://docs.docker.com/build/building/multi-stage/) -- [BuildKit](https://docs.docker.com/build/buildkit/) - -### Optimization Tools -- [dive - Docker image analyzer](https://github.com/wagoodman/dive) -- [hadolint - Dockerfile linter](https://github.com/hadolint/hadolint) -- [docker-slim](https://github.com/docker-slim/docker-slim) - ---- - -## Appendix A: Sample Optimized Dockerfile Structure - -```dockerfile -# syntax=docker/dockerfile:1.4 - -# ============================================================================== -# Stage 1: Base Builder -# ============================================================================== -FROM buildpack-deps:jammy AS base-builder -ENV DEBIAN_FRONTEND=noninteractive -RUN apt-get update && \ - apt-get install -y --no-install-recommends \ - ca-certificates \ - curl \ - git \ - wget && \ - rm -rf /var/lib/apt/lists/* - -# ============================================================================== -# Stage 2: MRtrix3 Builder -# ============================================================================== -FROM base-builder AS mrtrix3-builder -RUN apt-get update && \ - apt-get install -y --no-install-recommends \ - g++ \ - libeigen3-dev \ - libfftw3-dev \ - libgl1-mesa-dev \ - libpng-dev \ - libqt5opengl5-dev \ - libqt5svg5-dev \ - libtiff5-dev \ - python3 \ - qt5-qmake \ - qtbase5-dev \ - zlib1g-dev && \ - rm -rf /var/lib/apt/lists/* - -WORKDIR /opt/mrtrix3 -RUN --mount=type=cache,target=/root/.cache \ - git clone --depth 1 --branch 3.0.4 https://github.com/MRtrix3/mrtrix3.git . && \ - ./configure && \ - ./build -persistent -nopaginate && \ - rm -rf .git tmp - -# ============================================================================== -# Stage 3: Additional stages for other components... -# ============================================================================== - -# ============================================================================== -# Final Production Stage -# ============================================================================== -FROM itisfoundation/jupyter-math:2.0.9 AS production -LABEL maintainer="ordonez" - -# Copy only runtime components from builders -COPY --from=mrtrix3-builder /opt/mrtrix3 ${HOME}/mrtrix3 - -# ... continue with other components ... -``` - ---- - -## Appendix B: Migration Checklist - -- [ ] Back up current working image -- [ ] Create git branch for modernization -- [ ] Document current versions -- [ ] Research updates for all components -- [ ] Create optimized Dockerfile -- [ ] Test build process -- [ ] Validate image size reduction -- [ ] Run component tests -- [ ] Run integration tests -- [ ] Update documentation -- [ ] Create release notes -- [ ] Tag release -- [ ] Deploy to test environment -- [ ] User acceptance testing -- [ ] Production deployment - ---- - -**Document Version:** 1.0 -**Last Updated:** December 1, 2025 -**Next Review:** After Phase 1 completion diff --git a/README.md b/README.md index 776460f..8654660 100755 --- a/README.md +++ b/README.md @@ -12,9 +12,9 @@ This is the source code of the JupyterLab Medical Image Processing OSPARC service. It is mostly centered on MRI data, and contains the following packages: "[JupyterLab](https://jupyter.org/) with a variety of Medical Image Processing packages pre-installed, mostly centered on MRI data: - [FSL](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki), a comprehensive library of analysis tools for FMRI, MRI and DTI brain imaging data. - - [FreeSurfer](https://surfer.nmr.mgh.harvard.edu/), an open source neuroimaging toolkit for processing, analyzing, and visualizing human brain MR images- + - [FreeSurfer](https://surfer.nmr.mgh.harvard.edu/) version 7.4.1 (June 2023), an open source neuroimaging toolkit for processing, analyzing, and visualizing human brain MR images. - [MRtrix3](https://www.mrtrix.org/) provides a set of tools to perform various types of diffusion MRI analyses. - - [Spinal Cord Toolbox](https://spinalcordtoolbox.com/), a comprehensive, free and open-source set of command-line tools dedicated to the processing and analysis of spinal cord MRI data. + - [Synb0 Disco](https://github.com/MASILab/Synb0-DISCO#readme), for distortion correction of diffusion weighted MRI without reverse phase-encoding scans or field-maps. - Python packages like [nibabel](https://nipy.org/nibabel/), [pyvista](https://docs.pyvista.org/version/stable/), [fsleyes](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FSLeyes) and [cc3d](https://github.com/seung-lab/connected-components-3d#readme). @@ -23,36 +23,6 @@ Either single commands can be run, or full .sh files. ____ -## Service Modernization - -**Current Version:** 1.2.1 - -A comprehensive modernization plan is available in [`MODERNIZATION_PLAN.md`](./MODERNIZATION_PLAN.md). This plan covers: - -- **Current Software Versions:** Detailed inventory of all installed medical imaging software and their versions -- **Version Updates:** Investigation of newer versions and update strategies for all components -- **Dockerfile Optimization:** Multi-stage build implementation and layer optimization techniques -- **Image Size Reduction:** Strategies to reduce the Docker image size by ~15-20% -- **Build Time Optimization:** Techniques to reduce build time by ~20-30% -- **Implementation Roadmap:** Phased approach for modernization over 12 weeks -- **Testing & Validation:** Comprehensive testing strategy to ensure no regression - -### Quick Version Reference - -| Component | Current Version | Location in Dockerfile | -|-----------|----------------|------------------------| -| Base Image | jupyter-math:2.0.9 | Line 3 | -| MRtrix3 | 3.0.4 | Lines 23-27 | -| FreeSurfer | 6.0.0 | Lines 41-49 | -| FSL | Not pinned | Lines 57-63 | -| ANTs | 2.4.4 | Lines 105-107 | -| Synb0-DISCO | master (unpinned) | Lines 75-103 | -| Spinal Cord Toolbox | 4.2.1 (disabled) | Lines 112-115 | - -For detailed version information, update strategies, and optimization techniques, please refer to the [Modernization Plan](./MODERNIZATION_PLAN.md). - -____ - ## Information for developers of this **o2S2PARC** service Building the docker image: diff --git a/UPDATING_FREESURFER.md b/UPDATING_FREESURFER.md deleted file mode 100644 index 588be83..0000000 --- a/UPDATING_FREESURFER.md +++ /dev/null @@ -1,21 +0,0 @@ -## Plan: Update FreeSurfer from v6.0 to v7 or v8 - -Based on the research, upgrading FreeSurfer will require installation method changes. The current tar.gz extraction approach will still work for v7, but using Debian packages (`.deb`) is now preferred and handles dependencies automatically. FreeSurfer v8 strongly recommends package installation and requires significantly more RAM (32GB vs 16GB). - -### Steps - -1. **Choose target version** - FreeSurfer v7.4.1 recommended over v8 due to lower RAM requirements (16GB vs 32GB) unless container deployment can guarantee 32GB+ memory. - -2. **Update FreeSurfer installation block in [`Dockerfile`](Dockerfile:47-56)** - Replace `wget` tar.gz extraction with `.deb` package installation using `apt-get install`, update FTP URL to HTTPS download URL, and verify dependencies are automatically handled. - -3. **Update license handling** - For v7, keep copying freesurfer_license.txt to `${FREESURFER_HOME}/license.txt` (no change needed); for v8, optionally switch to `FS_LICENSE` environment variable pointing to `/root/license.txt` for cleaner multi-version support. - -4. **Verify environment variables remain unchanged** - All existing `FREESURFER_HOME`, `FSFAST_HOME`, `MINC_BIN_DIR`, `MNI_DIR`, `PERL5LIB`, and `PATH` settings are compatible across v6, v7, and v8. - -### Further Considerations - -1. **Version selection** - Choose v7.4.1 for compatibility with current 16GB RAM containers, or v8.1.0 if 32GB+ RAM is available and faster processing (1.8-2.5hrs vs 10+hrs for recon-all) is desired? - -2. **Fallback for v8 memory constraints** - If upgrading to v8 with limited RAM, add `ENV FS_V8_XOPTS=0` to revert to older, less memory-intensive recon-all behavior? - -3. **Testing downstream dependencies** - Verify that existing pipeline scripts in Fariba_full_pipeline and validation workflows are compatible with the chosen FreeSurfer version after upgrade? \ No newline at end of file From 79ef03dd7a74d734cf76db6c806198adf9778cac Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez <56032114+JavierGOrdonnez@users.noreply.github.com> Date: Wed, 3 Dec 2025 11:11:28 +0100 Subject: [PATCH 07/19] Update Dockerfile Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- Dockerfile | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Dockerfile b/Dockerfile index fefc394..7f5fb48 100755 --- a/Dockerfile +++ b/Dockerfile @@ -54,10 +54,10 @@ RUN apt-get update && apt-get install -y tcsh bc libgomp1 perl-modules \ COPY --from=freesurfer-source /usr/local/freesurfer /usr/local/freesurfer # Set up FreeSurfer environment variables ENV FREESURFER_HOME=/usr/local/freesurfer/7.4.1 \ - FSFAST_HOME=/usr/local/freesurfer/7.4.1/fsfast \ - MINC_BIN_DIR=/usr/local/freesurfer/7.4.1/mni/bin \ - MNI_DIR=/usr/local/freesurfer/7.4.1/mni \ - PERL5LIB=/usr/local/freesurfer/7.4.1/mni/share/perl5 + FSFAST_HOME=$FREESURFER_HOME/fsfast \ + MINC_BIN_DIR=$FREESURFER_HOME/mni/bin \ + MNI_DIR=$FREESURFER_HOME/mni \ + PERL5LIB=$FREESURFER_HOME/mni/share/perl5 ENV PATH=$FREESURFER_HOME/bin:$MINC_BIN_DIR:$PATH # Copy license file to FreeSurfer home (v7 method) COPY freesurfer_license.txt $FREESURFER_HOME/license.txt From 5f810ee1e71c9d1fb6ee4fe4002960631967b1a2 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 13:33:00 +0100 Subject: [PATCH 08/19] update ci-service-integration-library --- .github/workflows/check-image.yml | 6 +++--- Makefile | 5 +++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index 8f4bb78..6c738db 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -13,11 +13,11 @@ jobs: - name: Checkout repo content uses: actions/checkout@v3 - name: ooil version - uses: docker://itisfoundation/ci-service-integration-library:v2.0.10 + uses: docker://${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} with: args: ooil --version - name: Assemble docker compose spec - uses: docker://itisfoundation/ci-service-integration-library:v2.0.10 + uses: docker://${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} with: args: ooil compose - name: create new dir @@ -44,7 +44,7 @@ jobs: - name: check disk space before image build run: df -h - name: Build all images if multiple - uses: docker://itisfoundation/ci-service-integration-library:v2.0.10 + uses: docker://${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} with: args: docker compose build --quiet - name: print docker image info diff --git a/Makefile b/Makefile index da87b7a..e02ca50 100755 --- a/Makefile +++ b/Makefile @@ -5,13 +5,14 @@ SHELL = /bin/sh export DOCKER_IMAGE_NAME ?= jupyter-medimproc export DOCKER_IMAGE_TAG ?= 1.2.1 +export CI_SERVICE_INTEGRATION_LIBRARY ?= itisfoundation/ci-service-integration-library:v2.1.25 define _bumpversion # upgrades as $(subst $(1),,$@) version, commits and tags @docker run -it --rm -v $(PWD):/${DOCKER_IMAGE_NAME} \ -u $(shell id -u):$(shell id -g) \ - itisfoundation/ci-service-integration-library:v2.0.10 \ + ${CI_SERVICE_INTEGRATION_LIBRARY} \ sh -c "cd /${DOCKER_IMAGE_NAME} && bump2version --verbose --config-file $(1) $(subst $(2),,$@)" endef @@ -25,7 +26,7 @@ version-patch version-minor version-major: .bumpversion.cfg ## increases service compose-spec: ## runs ooil to assemble the docker-compose.yml file @docker run -it --rm -v $(PWD):/${DOCKER_IMAGE_NAME} \ -u $(shell id -u):$(shell id -g) \ - itisfoundation/ci-service-integration-library:v2.0.10 \ + ${CI_SERVICE_INTEGRATION_LIBRARY} \ sh -c "cd /${DOCKER_IMAGE_NAME} && ooil compose" .PHONY: build From 4b9188817abafc480ad009e27494b807473ff656 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 13:58:47 +0100 Subject: [PATCH 09/19] [CI] trying to fix GitHub CI --- .github/workflows/check-image.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index 6c738db..bf1e577 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -13,11 +13,11 @@ jobs: - name: Checkout repo content uses: actions/checkout@v3 - name: ooil version - uses: docker://${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} + uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY with: args: ooil --version - name: Assemble docker compose spec - uses: docker://${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} + uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY with: args: ooil compose - name: create new dir @@ -44,7 +44,7 @@ jobs: - name: check disk space before image build run: df -h - name: Build all images if multiple - uses: docker://${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} + uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY with: args: docker compose build --quiet - name: print docker image info From 674c341dffe57981ddc801761242a0f39b1a61db Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 13:59:58 +0100 Subject: [PATCH 10/19] [CI] keep trying --- .github/workflows/check-image.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index bf1e577..14716af 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -13,11 +13,11 @@ jobs: - name: Checkout repo content uses: actions/checkout@v3 - name: ooil version - uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY + uses: docker://${CI_SERVICE_INTEGRATION_LIBRARY} with: args: ooil --version - name: Assemble docker compose spec - uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY + uses: docker://${CI_SERVICE_INTEGRATION_LIBRARY} with: args: ooil compose - name: create new dir @@ -44,7 +44,7 @@ jobs: - name: check disk space before image build run: df -h - name: Build all images if multiple - uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY + uses: docker://${CI_SERVICE_INTEGRATION_LIBRARY} with: args: docker compose build --quiet - name: print docker image info From 892baa3a675827b1e66e3aaf595903aea0bdf7f4 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 14:02:12 +0100 Subject: [PATCH 11/19] [CI] trying again --- .github/workflows/check-image.yml | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index 14716af..6bed68e 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -1,5 +1,9 @@ name: Build and check image +env: + CI_SERVICE_INTEGRATION_LIBRARY: itisfoundation/ci-service-integration-library:v2.1.25 + + on: push: branches: @@ -13,11 +17,11 @@ jobs: - name: Checkout repo content uses: actions/checkout@v3 - name: ooil version - uses: docker://${CI_SERVICE_INTEGRATION_LIBRARY} + uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY with: args: ooil --version - name: Assemble docker compose spec - uses: docker://${CI_SERVICE_INTEGRATION_LIBRARY} + uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY with: args: ooil compose - name: create new dir @@ -44,7 +48,7 @@ jobs: - name: check disk space before image build run: df -h - name: Build all images if multiple - uses: docker://${CI_SERVICE_INTEGRATION_LIBRARY} + uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY with: args: docker compose build --quiet - name: print docker image info From f2c74cdaa3e4fce1512c0c6b12aceb9a34300daa Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 14:04:35 +0100 Subject: [PATCH 12/19] [FIX] revert to explicit declaration --- .github/workflows/check-image.yml | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index 6bed68e..ea0b959 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -1,9 +1,5 @@ name: Build and check image -env: - CI_SERVICE_INTEGRATION_LIBRARY: itisfoundation/ci-service-integration-library:v2.1.25 - - on: push: branches: @@ -17,11 +13,11 @@ jobs: - name: Checkout repo content uses: actions/checkout@v3 - name: ooil version - uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY + uses: docker://itisfoundation/ci-service-integration-library:v2.1.25 with: args: ooil --version - name: Assemble docker compose spec - uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY + uses: docker://itisfoundation/ci-service-integration-library:v2.1.25 with: args: ooil compose - name: create new dir @@ -48,7 +44,7 @@ jobs: - name: check disk space before image build run: df -h - name: Build all images if multiple - uses: docker://$CI_SERVICE_INTEGRATION_LIBRARY + uses: docker://itisfoundation/ci-service-integration-library:v2.1.25 with: args: docker compose build --quiet - name: print docker image info From 50e67021e3579cdc98cf497c7f126a3bdb99380e Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 14:07:49 +0100 Subject: [PATCH 13/19] [CI] centralize base builder image definitioin --- .github/workflows/check-image.yml | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index ea0b959..96811be 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -9,17 +9,17 @@ on: jobs: verify-image-build: runs-on: ubuntu-latest + env: + CI_SERVICE_INTEGRATION_LIBRARY: itisfoundation/ci-service-integration-library:v2.1.25 steps: - name: Checkout repo content uses: actions/checkout@v3 + - name: Pull CI service integration library + run: docker pull ${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} - name: ooil version - uses: docker://itisfoundation/ci-service-integration-library:v2.1.25 - with: - args: ooil --version + run: docker run --rm ${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} ooil --version - name: Assemble docker compose spec - uses: docker://itisfoundation/ci-service-integration-library:v2.1.25 - with: - args: ooil compose + run: docker run --rm ${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} ooil compose - name: create new dir run: sudo mkdir -p /mnt/docker - name: Maximize build space @@ -44,8 +44,6 @@ jobs: - name: check disk space before image build run: df -h - name: Build all images if multiple - uses: docker://itisfoundation/ci-service-integration-library:v2.1.25 - with: - args: docker compose build --quiet + run: docker run --rm ${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} docker compose build - name: print docker image info run: docker images From 6ff64b1a8a357a851c387b702a5eea57802bb0d3 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 14:11:24 +0100 Subject: [PATCH 14/19] [CI] revert to previous structure --- .github/workflows/check-image.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index 96811be..44fc771 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -17,9 +17,13 @@ jobs: - name: Pull CI service integration library run: docker pull ${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} - name: ooil version - run: docker run --rm ${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} ooil --version + uses: docker://itisfoundation/ci-service-integration-library:v2.1.25 + with: + args: ooil --version - name: Assemble docker compose spec - run: docker run --rm ${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} ooil compose + uses: docker://itisfoundation/ci-service-integration-library:v2.1.25 + with: + args: ooil compose - name: create new dir run: sudo mkdir -p /mnt/docker - name: Maximize build space From 7f647895ec25f8bc98fed6eef3f0ad1ed51e2e45 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 14:17:49 +0100 Subject: [PATCH 15/19] change v2.1.25 by v2.1.27 --- .github/workflows/check-image.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index 44fc771..8d67471 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -10,18 +10,18 @@ jobs: verify-image-build: runs-on: ubuntu-latest env: - CI_SERVICE_INTEGRATION_LIBRARY: itisfoundation/ci-service-integration-library:v2.1.25 + CI_SERVICE_INTEGRATION_LIBRARY: itisfoundation/ci-service-integration-library:v2.1.27 steps: - name: Checkout repo content uses: actions/checkout@v3 - name: Pull CI service integration library run: docker pull ${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} - name: ooil version - uses: docker://itisfoundation/ci-service-integration-library:v2.1.25 + uses: docker://itisfoundation/ci-service-integration-library:v2.1.27 with: args: ooil --version - name: Assemble docker compose spec - uses: docker://itisfoundation/ci-service-integration-library:v2.1.25 + uses: docker://itisfoundation/ci-service-integration-library:v2.1.27 with: args: ooil compose - name: create new dir From 27dd10481be1ab910e4f929d41f3e69fc29cea65 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 14:21:55 +0100 Subject: [PATCH 16/19] [CI] revert to original CI config with new image --- .github/workflows/check-image.yml | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index 8d67471..a52c839 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -9,13 +9,9 @@ on: jobs: verify-image-build: runs-on: ubuntu-latest - env: - CI_SERVICE_INTEGRATION_LIBRARY: itisfoundation/ci-service-integration-library:v2.1.27 steps: - name: Checkout repo content uses: actions/checkout@v3 - - name: Pull CI service integration library - run: docker pull ${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} - name: ooil version uses: docker://itisfoundation/ci-service-integration-library:v2.1.27 with: @@ -48,6 +44,8 @@ jobs: - name: check disk space before image build run: df -h - name: Build all images if multiple - run: docker run --rm ${{ env.CI_SERVICE_INTEGRATION_LIBRARY }} docker compose build + uses: docker://itisfoundation/ci-service-integration-library:v2.1.27 + with: + args: docker compose build --quiet - name: print docker image info - run: docker images + run: docker images \ No newline at end of file From bd6c3e4cb230bf7bf425bc03cbded66da61afe78 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 14:33:07 +0100 Subject: [PATCH 17/19] [CI] activating verbose in docker build for debugging --- .github/workflows/check-image.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index a52c839..74960ad 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -46,6 +46,6 @@ jobs: - name: Build all images if multiple uses: docker://itisfoundation/ci-service-integration-library:v2.1.27 with: - args: docker compose build --quiet + args: docker compose build --verbose - name: print docker image info run: docker images \ No newline at end of file From b9ea2bc96e27a14f013d93c5c2d64cbb9b56e893 Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 14:40:40 +0100 Subject: [PATCH 18/19] [CI] fix logging flag --- .github/workflows/check-image.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index 74960ad..cbaab21 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -11,7 +11,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repo content - uses: actions/checkout@v3 + uses: actions/checkout@v3 ## update?? up to v6 released - name: ooil version uses: docker://itisfoundation/ci-service-integration-library:v2.1.27 with: @@ -46,6 +46,6 @@ jobs: - name: Build all images if multiple uses: docker://itisfoundation/ci-service-integration-library:v2.1.27 with: - args: docker compose build --verbose + args: docker compose build --progress=plain - name: print docker image info run: docker images \ No newline at end of file From b5ad2554efdfb2ae8e16a710d8bd3addeaa5a4cb Mon Sep 17 00:00:00 2001 From: Javier Garcia Ordonez Date: Wed, 3 Dec 2025 14:41:56 +0100 Subject: [PATCH 19/19] dummy commit --- .github/workflows/check-image.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/check-image.yml b/.github/workflows/check-image.yml index cbaab21..1b6a4ed 100644 --- a/.github/workflows/check-image.yml +++ b/.github/workflows/check-image.yml @@ -11,7 +11,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repo content - uses: actions/checkout@v3 ## update?? up to v6 released + uses: actions/checkout@v3 - name: ooil version uses: docker://itisfoundation/ci-service-integration-library:v2.1.27 with: