GraphNN-VR: Graph Neural Network for VR Teaching System

A Graph Neural Network (GNN) based VR teaching system that enables collaborative control of virtual characters between teachers and students. The system dynamically adjusts weights to optimize student action following and improve action accuracy.

🎯 Project Overview

This project implements a real-time VR teaching system using Graph Neural Networks to predict teacher movements based on student input data. The system features dynamic weight adjustment that optimizes the balance between teacher guidance and student autonomy.

Key Features

• Real-time Prediction: Predicts teacher future positions and rotations based on student past 1-second (50 frames) data
• Dynamic Weight Adjustment: Automatically adjusts weights based on hand distance offset from baseline
• High Accuracy: Achieves ≥95% accuracy with <40ms inference delay
• Multi-model Support: Includes GNN, LSTM, and GCN baseline implementations
• Comprehensive Visualization: Trajectory comparison plots and ablation study heatmaps

📊 Data Specifications

Input Data Format

• Sampling Rate: 50Hz
• Frame Data: 32-dimensional tensor per frame
  • 0-5: Hand positions (6D) - [Left X, Y, Z, Right X, Y, Z]
  • 6-13: Hand rotations (8D) - [Left quaternion x,y,z,w, Right quaternion x,y,z,w]
  • 14-19: Hand velocities (6D) - [Left velocity X,Y,Z, Right velocity X,Y,Z]
  • 20-25: Current target ball distances (6D) - [Left to ball X,Y,Z, Right to ball X,Y,Z]
  • 26-31: Next target ball offsets (6D) - [Left offset X,Y,Z, Right offset X,Y,Z]

Data Normalization

• All positions, velocities, distances, and offsets are normalized by dividing by RADIUS (150cm)
• Euler angles are converted to quaternions in [x, y, z, w] order

Dataset Structure

data/
├── processed/          # Processed .pt files
│   ├── Data_CYJ_J/
│   │   ├── player_0_All.pt  # Teacher data
│   │   └── player_1_All.pt  # Student data
│   └── ... (25 datasets)
└── raw/               # Original JSON data
    ├── Data_CYJ_J/
    ├── Data_GWX_J/
    └── ... (25 participants)

🚀 Installation

Prerequisites

• Python 3.10+
• CUDA-compatible GPU (optional, for acceleration)

Using Poetry (Recommended)

# Install Poetry if not already installed
curl -sSL https://install.python-poetry.org | python3 -

# Clone the repository
git clone <repository-url>
cd GraphNN-VR-v5.2

# Install dependencies
poetry install

# Activate virtual environment
poetry shell

Using pip

# Clone the repository
git clone <repository-url>
cd GraphNN-VR-v5.2

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install torch torch-geometric numpy tqdm hydra-core matplotlib

📁 Project Structure

GraphNN-VR-v5.2/
├── graphnn/                    # Main package
│   ├── data/                   # Data processing modules
│   │   ├── dataset.py         # Dataset classes
│   │   ├── preprocess.py      # Data preprocessing
│   │   └── gen_mean_std.py    # Statistics generation
│   ├── train/                  # Training scripts
│   │   ├── gnn_vr_2.py       # Main GNN training
│   │   ├── improved_train_gnn_vr.py
│   │   ├── gru.py             # GRU baseline
│   │   └── lstm.py            # LSTM baseline
│   ├── engine/                 # Model and inference
│   │   ├── model.py           # Model architecture
│   │   ├── eval.py            # Evaluation metrics
│   │   └── infer_sliding_gnn.py  # Inference engine
│   ├── server/                 # Real-time server
│   │   ├── gnn_socket.py      # GNN inference server
│   │   └── ue_socket.py       # UE4 communication
│   ├── utils/                  # Utilities
│   │   ├── metrics.py         # Performance metrics
│   │   └── smooth.py          # Data smoothing
│   └── viz/                    # Visualization
│       ├── plot_traj_all.py   # Trajectory plotting
│       └── plot_convergence.py # Training convergence
├── data/                       # Data directory
├── docs/                       # Documentation
├── logs/                       # Training logs
├── outputs/                    # Model outputs
└── scripts/                    # Utility scripts

🎮 Usage

Data Preprocessing

# Preprocess raw data
python -m graphnn.data.preprocess

# Generate statistics for normalization
python -m graphnn.data.gen_mean_std

Training Models

GNN Model (Main)

# Train GNN model
python -m graphnn.train.gnn_vr_2

# Train improved GNN
python -m graphnn.train.improved_train_gnn_vr

Baseline Models

# Train LSTM baseline
python -m graphnn.train.lstm

# Train GRU baseline
python -m graphnn.train.gru

Evaluation

# Evaluate model performance
python -m graphnn.engine.eval

# Run inference with sliding window
python -m graphnn.engine.infer_sliding_gnn

Real-time Server

# Start GNN inference server
python -m graphnn.server.gnn_socket

# Start UE4 communication server
python -m graphnn.server.ue_socket

Visualization

# Plot trajectory comparisons
python -m graphnn.viz.plot_traj_all

# Plot training convergence
python -m graphnn.viz.plot_convergence

🎯 Performance Metrics

Metric	Target	Current
Accuracy	≥95%	✅
Inference Delay	<40ms	✅
Model Size	Optimized	✅

Baseline Comparison

• GNN: Primary model with graph-based architecture
• LSTM: Sequential baseline for comparison
• GCN: Graph Convolutional Network baseline

🔧 Configuration

The project uses Hydra for configuration management. Key configuration files are located in the training scripts and can be customized for different experiments.

Key Parameters

• Sequence Length: 50 frames (1 second)
• Prediction Horizon: Future positions and rotations
• Weight Adjustment: Dynamic based on offset from baseline
• Normalization Radius: 150cm

📈 Results

Trajectory Prediction

The system successfully predicts teacher movements with high accuracy, enabling smooth collaborative control in VR environments.

Weight Adjustment

Dynamic weight adjustment ensures optimal balance between teacher guidance and student autonomy based on performance metrics.

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• VR teaching system research team
• PyTorch Geometric community
• All contributors and testers

📞 Contact

For questions and support, please open an issue on GitHub or contact the development team.

---

Note: This project is designed for VR teaching applications and requires appropriate VR hardware and software for full functionality.