Flowchart of the proposed SORA framework.
Real-image example of keypoint reduction and role-aware interaction-region construction.
Structured event output and query-based review interface.
Confusion matrix of the proposed SORA model on the 807-sample test set.
[SORA：結合骨架引導視覺語言模型之嚴重度感知序位風險評估方法於嬰幼兒照護之研究]
Infant-care monitoring requires graded risk decisions rather than binary anomaly labels because unsafe situations differ in severity and response urgency. This thesis proposes SORA, a severity-aware ordinal risk assessment framework that integrates Contrastive Language-Image Pre-training (CLIP)-based visual encoding, skeleton-guided caregiver-infant interaction cues, Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (BLIP-2) captions, and keyword-enhanced text encoding. Instead of treating the three labels as independent nominal classes, SORA predicts a scalar risk score so that low-, medium-, and high-risk states remain ordered along one response axis and are decoded by validation-selected thresholds. It also stores captions, interaction cues, timestamps, and risk labels as structured event records for later review. On a three-level proxy benchmark, SORA achieves 89.22% accuracy, 88.70% Macro-F1, and 99.38% high-risk recall, improving over AnomalyCLIP by 4.71, 4.88, and 11.11 percentage points, respectively. The results show that ordered risk modeling and interaction-aware multimodal evidence reduce safety-critical underestimation while preserving interpretable review records.
SUMMARY (中文總結):
自動化監測系統逐漸採用異常偵測方法，以降低安全監測中對人工監督的依賴。然而，多數既有方法將任務定義為正常與異常之間的二元判斷，難以適用於嬰幼兒照護情境。嬰幼兒照護中的不安全狀況具有不同嚴重程度，並需要不同層級的照護回應。因此，本研究的核心問題並非單純的異常偵測，而是能夠反映風險嚴重程度與回應優先順序的多層級風險評估。
本論文提出多模態嬰幼兒照護風險評估框架SORA，整合CLIP視覺特徵、骨架引導之照護者與嬰幼兒互動線索、BLIP-2場景描述及語意關鍵詞。模型輸出單一風險分數，並以驗證集選定的兩個有序門檻判定低、中、高風險。訓練時採用安全導向的序位學習、成對排序及向下誤判懲罰，以提升相鄰風險層級的區辨能力，並降低高風險情境遭嚴重低估的情形。同時，系統將風險標籤、時間戳記、場景描述、骨架互動描述及關鍵詞保存為結構化事件紀錄，以支援後續查詢與事件回顧。
本研究使用公開嬰幼兒照護影像、成人表情與暴力互動資料建立三層級代理基準進行評估。實驗結果顯示，SORA的整體分類準確率達89.22%，Macro-F1達88.70%，高風險召回率達99.38%；相較於AnomalyCLIP，分別提升4.71、4.88及11.11個百分點。結果顯示，序位風險建模與互動感知多模態證據可提升嬰幼兒照護風險分級能力，並降低安全關鍵情境遭低估的情形。
RESULTS:
Comparison with baseline risk assessment methods.
| Method | CES ↑ | High Recall ↑ | Macro-F1 ↑ | Accuracy ↑ | QWK ↑ |
|---|---|---|---|---|---|
| CLIP | 0.7133 | 0.7901 | 0.7077 | 0.7224 | 0.6087 |
| AnomalyCLIP | 0.8499 | 0.8827 | 0.8382 | 0.8451 | 0.7610 |
| AnomalyCLIP (ordinal) | 0.8635 | 0.9259 | 0.8472 | 0.8550 | 0.7922 |
| SORA (multi-logit) | 0.8617 | 0.9567 | 0.7967 | 0.8017 | 0.8368 |
| SORA (ordinal) | 0.9153 | 0.9938 | 0.8870 | 0.8922 | 0.8630 |
The full ordinal SORA model achieves the best CES, high-risk recall, Macro-F1, accuracy, and QWK among the baseline risk assessment systems compared above. Among 162 high-risk test samples, SORA correctly identifies 161, leaving only one high-risk underestimation.
REFERENCES:
[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision," in Proceedings of the 38th International Conference on Machine Learning, vol. 139, pp. 8748-8763, 2021.
[2] Q. Zhou, G. Pang, Y. Tian, S. He, and J. Chen, "AnomalyCLIP: Object-agnostic prompt learning for zero-shot anomaly detection," in The Twelfth International Conference on Learning Representations, 2024.
[3] W. Cao, V. Mirjalili, and S. Raschka, "Rank consistent ordinal regression for neural networks with application to age estimation," Pattern Recognition Letters, vol. 140, pp. 325-331, 2020.
[4] J. Li, D. Li, S. Savarese, and S. Hoi, "BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," in Proceedings of the 40th International Conference on Machine Learning, vol. 202, pp. 19730-19742, 2023.
[5] T. Jiang, X. Xie, and Y. Li, "RTMW: Real-time multi-person 2D and 3D whole-body pose estimation," arXiv preprint arXiv:2407.08634, 2024.
[6] C. Lyu, W. Zhang, H. Huang, Y. Zhou, Y. Wang, Y. Liu, S. Zhang, and K. Chen, "RTMDet: An empirical study of designing real-time object detectors," arXiv preprint arXiv:2212.07784, 2022.
[7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems, vol. 30, pp. 6000-6010, 2017.