Overall two-stage framework
Overall pipeline of the proposed two-stage material recognition framework.
Stage 1 architecture
Stage 1 uses a T2T-ViT encoder and Reverse T2T decoder with skip connections to produce multi-scale material logit maps.
Stage 2 architecture
Stage 2 integrates Stage 1 material logits, RGB appearance, and object-aware cues derived from Ultralytics YOLO26.
Deep Sets global context encoding
Deep Sets-based global object-context encoding converts a variable-size unordered detection set into a fixed-size permutation-invariant representation.
Input image and final prediction
Representative vehicle image and corresponding final material prediction.
Confusion matrices
Row-normalized confusion matrices of Stage 1 and Stage 1 + Stage 2 on LMD.

A Two-Stage Framework for Material Recognition:
Pixel-Level Analysis with Object-Awareness

[一種材質識別的兩階段框架：具有物件感知的像素級分析]

ABSTRACT:
Material recognition is crucial in computer vision applications such as robotics, autonomous driving, and extended reality. Traditional methods primarily rely on texture and color features, but they perform poorly when different materials share similar textures or when the same material appears differently under varying conditions. To address this challenge, some studies have attempted to directly incorporate object information in a single stage. However, this direct fusion often interferes with pixel-level details and increases model complexity. Consequently, a multi-stage approach provides a more flexible and interpretable alternative. To overcome these issues, we propose a two-stage method for material recognition that integrates object awareness into pixel-level analysis. In the first stage, we adopt a Tokens-to-Token Vision Transformer encoder and a Reverse Tokens-to-Token decoder to capture fine-grained texture features. In the second stage, we introduce object context obtained via Ultralytics YOLO26 detection and fuse it with the first-stage outputs and the original RGB image. We evaluated our method on the Local Materials Database and compared it with existing state-of-the-art approaches. Experimental results show that our method achieves first or second place in most material categories and surpasses existing methods in overall accuracy and mean class performance.

SUMMARY (中文總結):
材質識別在機器人、自動駕駛與延展實境等電腦視覺應用中具有重要作用。然而，傳統方法主要依賴紋理與顏色特徵進行材質識別。 當不同材質具有相似的紋理，或相同材質在不同條件下呈現不同外觀時，這些方法的辨識表現容易下降。為了解決此問題，部分研究嘗試在單一階段中直接融合物件資訊。 然而，這種直接融合方式可能干擾像素層級的細節、增加模型複雜度，並降低整體效能。本研究提出一種兩階段材質識別方法，將物件感知資訊整合至像素層級的材質分析中。 在第一階段中，模型著重於擷取像素層級的材質特徵。在第二階段中，模型引入透過 Ultralytics YOLO26 物件偵測所獲得的物件資訊， 並將其與第一階段的預測結果及原始輸入影像結合，以進一步修正材質預測。實驗結果顯示，所提出的方法在大多數材質類別中取得第一或第二名的表現， 並在整體準確率與平均類別準確率上超越現有方法。

RESULTS:
On the Local Materials Database (LMD), Stage 1 achieves a pixel accuracy (PA) of 88.44% and a mean class accuracy (mCA) of 83.55%. After object-aware refinement, the complete Stage 1 + Stage 2 framework reaches 91.07% PA and 87.38% mCA. Stage 2 improves all 16 material categories. The largest improvement is observed for ceramic (+10.70 percentage points), followed by stone (+8.09), concrete (+7.61), metal (+6.61), and rubber (+6.41).

Comparison with material segmentation methods reported in the thesis.

Method Pixel Accuracy (%) ↑ Mean Class Accuracy (%) ↑
ResNet-15280.6873.87
ResNeSt-10182.4575.31
EfficientNet-B583.1776.91
Swin-t84.7179.06
CAM-SegNet-DBA86.1279.85
DBAT86.8581.05
Our (Stage 1 only)88.4483.55
Our (Stage 1 + Stage 2)91.0787.38

Note: The baseline values above are literature-reported results cited in the thesis rather than results reproduced under an identical training environment. The table is therefore intended as a literature-based comparison.

The additional DMS evaluation further shows that Stage 2 improves PA from 64.05% to 71.92% and mCA from 42.93% to 47.58%, indicating that the refinement mechanism remains effective under a different material taxonomy and annotation setting.

REFERENCES:

  • G. Schwartz and K. Nishino, “Recognizing material properties from images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 8, pp. 1981-1995, 2020. doi: 10.1109/TPAMI.2019.2907850.
  • Y. Heng, S. Dasmahapatra, and H. Kim, “DBAT: Dynamic backward attention transformer for material segmentation with cross-resolution patches,” 2024. doi: 10.48550/arXiv.2305.03919.
  • L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z.-H. Jiang, F. E. H. Tay, J. Feng, and S. Yan, “Tokens-to-Token ViT: Training vision transformers from scratch on ImageNet,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 538-547, 2021. doi: 10.1109/ICCV48922.2021.00060.
  • N. Liu, N. Zhang, K. Wan, L. Shao, and J. Han, “Visual Saliency Transformer,” in IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4702-4712, 2021. doi: 10.1109/ICCV48922.2021.00468.
  • G. Jocher, J. Qiu, M. Liu, S. Lyu, F. C. Akyon, and M. E. Kalfaoglu, “Ultralytics YOLO26: Unified real-time end-to-end vision models,” 2026. doi: 10.48550/arXiv.2606.03748.
  • M. Zaheer, S. Kottur, S. Ravanbakhsh, B. P?czos, R. Salakhutdinov, and A. J. Smola, “Deep Sets,” in Advances in Neural Information Processing Systems (NeurIPS), pp. 3391-3401, 2017. doi: 10.48550/arXiv.1703.06114.
  • T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll?r, and C. L. Zitnick, “Microsoft COCO: Common Objects in Context,” in European Conference on Computer Vision (ECCV), pp. 740-755, 2014. doi: 10.1007/978-3-319-10602-1_48.