School of Computing and Data Sciences
FLAME University
Optical Music Recognition (OMR) systems have traditionally focused on homogeneous music notation, limiting their applicability to real-world scenarios where multiple notation styles coexist within the same document. This work introduces SWARA (Sheet music With Augmented Recognition for metAverse), a unified framework for heterogeneous music notation recognition and real-time symbolic-to-immersive mapping. Our contributions include: (1) the first annotated percussion sheet music dataset integrated with piano notation, (2) a unified recognition pipeline capable of detecting heterogeneous music symbols and generating structured MusicXML, (3) a semantic Symbol-to-XR mapping framework with a novel evaluation metric for immersive music interaction, and (4) a collaborative metaverse environment that enables multi-user music learning and performance. By bridging the gap between static sheet music and interactive Extended Reality (XR), SWARA establishes a benchmark and an end-to-end Human-Document Interaction (HDI) framework for real-time heterogeneous music understanding.
Figure 1.Collaborative Human Document Interaction Framework : Symbolic-to-Immersive Mapping.
Figure 2. Overview of the proposed SWARA heterogeneous music notation dataset consisting of annotated percussion scores, staff-line masks, strip-wise annotations, and structured JSON labels.
All models were trained for 50 epochs on an NVIDIA RTX A4500 GPU using the Adam optimizer. The unified heterogeneous dataset consists of 49 musical symbol classes from piano and percussion notation with an 80:10:10 train-validation-test split. We evaluate both the proposed SWARA Staff Module for staff-line segmentation and multiple object detectors for heterogeneous musical symbol detection.
| Method | Dice | mIoU | Runtime |
|---|---|---|---|
| OEMER | 0.83 | 0.79 | 95 ms |
| U-Net Baseline | 0.87 | 0.84 | 54 ms |
| SWARA Staff Module | 0.93 | 0.90 | 41 ms |
| Detector | mAP50 | mAP50:95 | Precision | Recall |
|---|---|---|---|---|
| YOLOv11-l | 0.92 | 0.80 | 0.93 | 0.86 |
| YOLOv11-m | 0.90 | 0.77 | 0.92 | 0.84 |
| YOLOv11-s | 0.87 | 0.72 | 0.90 | 0.82 |
| Faster R-CNN | 0.87 | 0.72 | 0.89 | 0.82 |
(a) Percussion notation detection demonstrating accurate localization of heterogeneous musical symbols.
(b) Piano notation detection showing robust symbol recognition across complex musical scores.
The robustness of the proposed framework is evaluated under challenging image degradations, including Gaussian blur, additive noise, geometric rotation, severe downsampling, and GAN-based image enhancement. The qualitative results demonstrate reliable symbol localization and stable performance across diverse real-world imaging conditions.
Figure 5. Qualitative robustness evaluation of the proposed framework under diverse image perturbations, including Gaussian blur, additive noise, rotation, low-resolution inputs, and GAN-based image enhancement. The proposed detector consistently localizes heterogeneous musical symbols across challenging imaging conditions, demonstrating strong resilience to appearance and geometric variations.