Project • Paper • HuggingFace • Overview • Quickstart • Citation
- 2025-11-01, Our paper has been ACCEPTED for publication as a REGULAR paper in the IEEE TASLP(Transactions on Audio, Speech and Language Processing).
- 2025-02-28, We release the paper.
Get training data and test data from HuggingFace: https://huggingface.co/datasets/zjunlp/Knowledge2Data
git clone https://github.com/zjunlp/Knowledge2Data
cd Knowledge2Data
conda create -n skg python==3.9
conda activate skg
pip install -r requirements.txt
| 🎯 Model Name | 🤗 HuggingFace |
|---|---|
| Diffusers-generation-text-box | gligen/diffusers-generation-text-box |
| Sam-vit-base | stabilityai/stable-diffusion-xl-refiner-1.0 |
| Stable-diffusion-xl-refiner | facebook/sam-vit-base |
cd src
export OPENAI_API_KEY="YOUR_API_KEY"export SKG_HF_MODELS="LOCAL_HUGGINGFACE_MODELS_DIR"sh run_skg.shYou can also customize objects and their spatial relationships to form Spatial KG. Save the file format as a JSON file similar to "src/data/skg_demo.json".
sh run_data.shFor custom data, only the input file parameters "--input_file" need to be modified.
You can find generated data in "src/data" and images in "src/img_generations" as default. If you want to generate more data, you can modify the parameters including "--num_scenes" (generate_scenes.py) and "--repeats" (generate_images.py).
This project is based on open-source projects including LLM-groundedDiffusion. Thanks for their great contributions!
Please cite the following paper if you use this project in your work.
@misc{xue2025spatialknowledgegraphguidedmultimodal,
title={Spatial Knowledge Graph-Guided Multimodal Synthesis}, author={Yida Xue and Zhen Bi and Jinnan Yang and Jungang Lou and Huajun Chen and Ningyu Zhang},
year={2025},
eprint={2505.22633},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2505.22633}, }