RESEARCH · RESEARCH · #518
Fine-tuned Stable Diffusion XL and LLaMA enable controllable Ulos motif generation
arXiv:2609.17987v1 presents a multimodal generative framework that fine-tunes Stable Diffusion XL v1.0 via LoRA and integrates a multimodal LLaMA 1.5-7B to generate culturally faithful Batak Ulos motifs. The system uses four conditioning mechanisms (text, image, representation, semantic map/ControlNet); an ablation across three scenarios found Text+Image+Semantic Map achieved the best FID (270) and CLIP (0.65–0.70) but lowest SSIM (0.65), Text+Image+Representation gave the best overall balance (SSIM 0.84, FID 280), and combining all four worsened FID to 330; qualitative evaluation with nine weavers and 30 public participants showed statistically significant positive acceptance (Wilcoxon p=0.007 and p<0.001), and a web prototype for text- and image-to-image generation was developed.
KEY POINTS
- arXiv:2609.17987v1 presents a multimodal generative framework that fine-tunes Stable Diffusion XL v1.0 via LoRA and integrates a multimodal LLaMA 1.5-7B to generate culturally faithful Batak Ulos motifs.
- The system uses four conditioning mechanisms (text, image, representation, semantic map/ControlNet); an ablation across three scenarios found Text+Image+Semantic Map achieved the best FID (270) and CLIP (0.65–0.70) but lowest SSIM (0.65), Text+Image+Representation gave the best overall balance (SSIM 0.84, FID 280), and combining all four worsened FID to 330; qualitative evaluation with nine weavers and 30 public participants showed statistically significant positive acceptance (Wilcoxon p=0.007 and p<0.001), and a web prototype for text- and image-to-image generation was developed.
- The work demonstrates practical multimodal conditioning trade-offs when fine-tuning large diffusion models for culturally specific design, offering a deployable tool and empirical guidance for preserving and digitizing heritage patterns.
WHY IT MATTERS
The work demonstrates practical multimodal conditioning trade-offs when fine-tuning large diffusion models for culturally specific design, offering a deployable tool and empirical guidance for preserving and digitizing heritage patterns.