Implement database synchronization for backtests, plus some new backtest features such as notes, and strategy snapshots

- Streamlined the process of preparing chart data by directly assigning formatted values to the chart_data dictionary.
- Removed redundant variable assignments to enhance code clarity and maintainability.

Add backtest session management features

- Introduced new API endpoints for managing backtest sessions, including fetching all sessions, retrieving a session by ID, updating session state, and removing sessions.
- Created a new BacktestSession model to handle session data in the database.
- Implemented functions for storing, updating, and deleting backtest sessions.
- Enhanced the backtest execution process to log session status and exceptions in the database.
- Updated transformers to format backtest session data for API responses.

Add clean_infinite_values function and enhance backtest session API

- Introduced clean_infinite_values function to recursively clean infinite values from data structures, replacing them with None or 0.
- Updated get_backtest_sessions endpoint to support pagination through request body.
- Modified get_backtest_sessions function to accept limit and offset parameters for improved session retrieval.
- Enhanced data transformation in get_backtest_session_for_load_more to clean infinite values from metrics, equity curve, trades, hyperparameters, and chart data.
- Added GetBacktestSessionsRequestJson model for structured request handling in the API.

Update backtest session status on cancellation request

- Added functionality to update the status of a backtest session to 'cancelled' when a cancellation request is processed.
- Enhanced the backtest_controller to import and utilize the update_backtest_session_status function for managing session states.

Add backtest session notes management and enhance session model

- Introduced a new API endpoint to update notes (title, description, strategy codes) for backtest sessions.
- Enhanced the BacktestSession model to include fields for user notes.
- Updated the update_backtest_session_results function to handle strategy codes.
- Modified transformers to include notes in the session data returned by the API.
- Added a new request model for updating backtest session notes.

Enhance backtest session retrieval with additional filters

- Updated the get_backtest_sessions function to support title search, status filtering, and date filtering for improved session management.
- Modified the GetBacktestSessionsRequestJson model to include new optional parameters for filtering.
- Enhanced the backtest_controller to utilize the updated session retrieval logic with pagination and filters.

Add purge sessions functionality to backtest controller and model

- Introduced a new API endpoint to purge backtest sessions older than a specified number of days.
- Implemented the purge_backtest_sessions function in the BacktestSession model to handle the deletion of old sessions based on the provided criteria.
- Enhanced error handling and logging for the purging process to improve debugging and user feedback.

Remove debug print statement from purge_backtest_sessions function in BacktestSession model to streamline error handling and improve logging clarity.

Add chart data retrieval endpoint for backtest sessions

- Introduced a new API endpoint to fetch chart data for a specific backtest session.
- Implemented validation for authorization tokens and session existence.
- Enhanced data transformation to clean infinite values from chart data before returning it.
- Updated the backtest session model to indicate the presence of chart data without returning it directly in the load more function.

Author: saleh-mir

jesse/controllers/backtest_controller.py

@@ -1,11 +1,19 @@
 from typing import Optional
-from fastapi import APIRouter, Header, Query
+from fastapi import APIRouter, Header, Query, Body
 from fastapi.responses import JSONResponse, FileResponse
-
+import json
 from jesse.services import auth as authenticator
 from jesse.services.multiprocessing import process_manager
-from jesse.services.web import BacktestRequestJson, CancelRequestJson
+from jesse.services.web import BacktestRequestJson, CancelRequestJson, UpdateBacktestSessionStateRequestJson, GetBacktestSessionsRequestJson, UpdateBacktestSessionNotesRequestJson
 import jesse.helpers as jh
+from jesse.models.BacktestSession import (
+    get_backtest_sessions as get_sessions,
+    update_backtest_session_state,
+    update_backtest_session_notes,
+    delete_backtest_session,
+    get_backtest_session_by_id as get_backtest_session_by_id_from_db
+)
+from jesse.services.transformers import get_backtest_session, get_backtest_session_for_load_more
 
 router = APIRouter(prefix="/backtest", tags=["Backtest"])
 
@@ -53,6 +61,9 @@ def cancel_backtest(request_json: CancelRequestJson, authorization: Optional[str
         return authenticator.unauthorized_response()
 
     process_manager.cancel_process(request_json.id)
+    
+    from jesse.models.BacktestSession import update_backtest_session_status
+    update_backtest_session_status(request_json.id, 'cancelled')
 
     return JSONResponse({'message': f'Backtest process with ID of {request_json.id} was requested for termination'},
                         status_code=202)
@@ -93,23 +104,158 @@ def download_backtest_log(session_id: str, token: str = Query(...)):
         return JSONResponse({'error': str(e)}, status_code=500)
 
 
+@router.post("/sessions")
+def get_backtest_sessions(request_json: GetBacktestSessionsRequestJson = Body(default=GetBacktestSessionsRequestJson()), authorization: Optional[str] = Header(None)):
+    """
+    Get a list of backtest sessions sorted by most recently updated with pagination
+    """
+    if not authenticator.is_valid_token(authorization):
+        return authenticator.unauthorized_response()
 
-@router.get("/logs/{session_id}")
-def get_logs(session_id: str, token: str = Query(...)):
+    # Get sessions from the database with pagination and filters
+    sessions = get_sessions(
+        limit=request_json.limit, 
+        offset=request_json.offset,
+        title_search=request_json.title_search,
+        status_filter=request_json.status_filter,
+        date_filter=request_json.date_filter
+    )
+
+    # Transform the sessions using the transformer
+    transformed_sessions = [get_backtest_session(session) for session in sessions]
+
+    return JSONResponse({
+        'sessions': transformed_sessions,
+        'count': len(transformed_sessions)
+    })
+
+
+@router.post("/sessions/{session_id}")
+def get_backtest_session_by_id(session_id: str, authorization: Optional[str] = Header(None)):
     """
-    Get logs as text for a specific session. Similar to download but returns text content instead of file.
+    Get a single backtest session by ID
     """
-    if not authenticator.is_valid_token(token):
+    if not authenticator.is_valid_token(authorization):
         return authenticator.unauthorized_response()
 
-    try:
-        from jesse.modes.data_provider import get_backtest_logs
-        content = get_backtest_logs(session_id)
+    # Get the session from the database
+    session = get_backtest_session_by_id_from_db(session_id)
 
-        if content is None:
-            return JSONResponse({'error': 'Log file not found'}, status_code=404)
+    if not session:
+        return JSONResponse({
+            'error': f'Session with ID {session_id} not found'
+        }, status_code=404)
 
-        return JSONResponse({'content': content}, status_code=200)
-    except Exception as e:
-        return JSONResponse({'error': str(e)}, status_code=500)
+    # Transform the session using the transformer
+    transformed_session = get_backtest_session_for_load_more(session)
+
+    return JSONResponse({
+        'session': transformed_session
+    })
+
+
+@router.post("/update-state")
+def update_session_state(request_json: UpdateBacktestSessionStateRequestJson, authorization: Optional[str] = Header(None)):
+    """
+    Update the state of a backtest session
+    """
+    if not authenticator.is_valid_token(authorization):
+        return authenticator.unauthorized_response()
+
+    update_backtest_session_state(request_json.id, request_json.state)
+
+    return JSONResponse({
+        'message': 'Backtest session state updated successfully'
+    })
+
+
+@router.post("/sessions/{session_id}/remove")
+def remove_backtest_session(session_id: str, authorization: Optional[str] = Header(None)):
+    """
+    Remove a backtest session from the database
+    """
+    if not authenticator.is_valid_token(authorization):
+        return authenticator.unauthorized_response()
+
+    session = get_backtest_session_by_id_from_db(session_id)
+
+    if not session:
+        return JSONResponse({
+            'error': f'Session with ID {session_id} not found'
+        }, status_code=404)
+
+    # Delete the session from the database
+    result = delete_backtest_session(session_id)
+
+    if not result:
+        return JSONResponse({
+            'error': f'Failed to delete session with ID {session_id}'
+        }, status_code=500)
+
+    return JSONResponse({
+        'message': 'Backtest session removed successfully'
+    })
+
+
+@router.post("/sessions/{session_id}/notes")
+def update_session_notes(session_id: str, request_json: UpdateBacktestSessionNotesRequestJson, authorization: Optional[str] = Header(None)):
+    """
+    Update the notes (title, description, strategy_codes) of a backtest session
+    """
+    if not authenticator.is_valid_token(authorization):
+        return authenticator.unauthorized_response()
+
+    session = get_backtest_session_by_id_from_db(session_id)
+
+    if not session:
+        return JSONResponse({
+            'error': f'Session with ID {session_id} not found'
+        }, status_code=404)
+
+    update_backtest_session_notes(session_id, request_json.title, request_json.description, request_json.strategy_codes)
+
+    return JSONResponse({
+        'message': 'Backtest session notes updated successfully'
+    })
+
+
+@router.post("/purge-sessions")
+def purge_sessions(request_json: dict = Body(...), authorization: Optional[str] = Header(None)):
+    """
+    Purge backtest sessions older than specified days
+    """
+    if not authenticator.is_valid_token(authorization):
+        return authenticator.unauthorized_response()
+    
+    days_old = request_json.get('days_old', None)
+    
+    from jesse.models.BacktestSession import purge_backtest_sessions
+    deleted_count = purge_backtest_sessions(days_old)
+    
+    return JSONResponse({
+        'message': f'Successfully purged {deleted_count} session(s)',
+        'deleted_count': deleted_count
+    }, status_code=200)
+
+
+@router.post("/sessions/{session_id}/chart-data")
+def get_backtest_session_chart_data(session_id: str, authorization: Optional[str] = Header(None)):
+    """
+    Get chart data for a specific backtest session
+    """
+    if not authenticator.is_valid_token(authorization):
+        return authenticator.unauthorized_response()
+    
+    session = get_backtest_session_by_id_from_db(session_id)
+    
+    if not session:
+        return JSONResponse({
+            'error': f'Session with ID {session_id} not found'
+        }, status_code=404)
+    
+    chart_data = jh.clean_infinite_values(json.loads(session.chart_data)) if session.chart_data else None
+    
+    return JSONResponse({
+        'chart_data': chart_data
+    })
 

jesse/helpers.py

@@ -1123,6 +1123,28 @@ def clear_output():
         click.clear()
 
 
+def clean_infinite_values(obj):
+    """
+    Recursively clean infinite values (inf, -inf) from data structures
+    by replacing them with None or 0
+    
+    :param obj: The object to clean (can be dict, list, or primitive)
+    :return: The cleaned object with infinite values replaced
+    """
+    import math
+    
+    if isinstance(obj, dict):
+        return {k: clean_infinite_values(v) for k, v in obj.items()}
+    elif isinstance(obj, list):
+        return [clean_infinite_values(item) for item in obj]
+    elif isinstance(obj, float):
+        if math.isinf(obj):
+            return None
+        return obj
+    else:
+        return obj
+
+
 def get_class_name(cls):
     # if it's a string, return it
     if isinstance(cls, str):