CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentation
Ziyang Gong 1∗, Zhixiang Wei 2∗, Di Wang 3∗, Xiaoxing Hu8∗, Xianzheng Ma3, Hongruixuan Chen 4,5, Yuru Jia 6,7, Yupeng Deng 9, Zhenming Ji10†, Xiangwei Zhu8†, Xue Yang1†, Naoto Yokoya4,5, Jing Zhang3, Bo Du3, Junchi Yan1, Liangpei Zhang3
1 SJTU 2 USTC 3 WHU 4 U Tokyo 5 RIKEN 6 KU Leuven 7 KTH 8 BIT 9 SYSU 10 UCAS
* Equal contribution. † Corresponding author.
[2024/11/14] We are releasing the benchmark collection. Click here to get illustrations of benchmarks!
[2024/11/06] The most checkpoints have been uploaded and you can access them in the huggingface badges.
The environment and inference steps please refer to the following installation. The inference codes and weights will be coming soon.
The benchmark collection in the paper is releasing and you can access it at here.
🎉🎉🎉 CrossEarth is the first VFM for Remote Sensing Domain Generalization (RSDG) semantic segmentation. We just release the arxiv paper of CrossEarth. You can access CrossEarth at here.
In Radar figure:
- CrossEarth achieves SOTA performances on 23 evaluation benchmarks across various segmentation scenes, demonstrating strong generalizability.
In UMAP figures:
CrossEarth extracts features that cluster closely for the same class across different domains, forming well-defined groups in feature space, demonstrating its ability to learn robust, domain-invariant features.
Moreover, CrossEarth features exhibit high inter-class separability, forming unique clusters for each class and underscoring its strong representational ability to distingguish different categories.
conda create -n CrossEarth -y
conda activate CrossEarth
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia -y
pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0"
pip install "mmsegmentation>=1.0.0"
pip install "mmdet>=3.0.0"
pip install xformers=='0.0.20' pip install -r requirements.txt
pip install future tensorboardFirst, download the model weights from the huggingface or Baidu Netdisk in the above badges. Notably, the checkpoints of dinov2_converted.pth and dinov2_converted_1024x1024.pth are needed for inference. Please download them and put them in the CrossEarth/checkpoints folder.
Second, change the file path in experiment config files (configs/base/datasets/xxx.py and configs/CrossEarth_dinov2/xxx.py), and run the following command to inference. (Take 512x512 inference as an example)
python tools/test.py configs/CrossEarth_dinov2/CrossEarth_dinov2_mask2former_512x512_bs1x4.py ./checkpoints/xxx.pthNotably, save path of pseudo labels is in the experiment config file. When testing CrossEarth on different benchmarks, you also need to change the class number in CrossEarth_dinov2_mask2former.py file.
Coming soon.
If you find CrossEarth helpful, please consider giving this repo a ⭐ and citing:
@article{crossearth,
title={CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentation},
author={Gong, Ziyang and Wei, Zhixiang and Wang, Di and Ma, Xianzheng and Chen, Hongruixuan and Jia, Yuru and Deng, Yupeng and Ji, Zhenming and Zhu, Xiangwei and Yokoya, Naoto and Zhang, Jing and Du, Bo and Zhang, Liangpei},
journal={arXiv preprint arXiv:2410.22629},
year={2024}
}