GLM-Image 是一种图像生成模型采用混合自回归扩散解码器架构。近日智谱开源了首个工业级离散自回归图像生成模型 GLM-ImageGLM-Image 采用混合架构将自回归模块与扩散解码器相结合。自回归部分部分基于 GLM-4-9B-0414 并由其初始化该模型拥有 90 亿个参数而扩散解码器则遵循 CogView4 的思路采用单流 DiT 结构拥有 70 亿个参数。在整体图像生成质量方面GLM-Image 与主流的潜在扩散方法相符但在文本渲染和知识密集型生成场景中展现出显著优势。它在需要精确语义理解和复杂信息表达的任务中表现尤为出色同时保持了高保真度和细粒度细节生成的强大能力。除了文本到图像的生成之外GLM-Image 还支持丰富的图像到图像任务包括图像编辑、风格迁移、身份保持生成和多主体一致性等。效果展示文生图图生图相关链接博客https://docs.z.ai/guides/image/glm-image主页https://z.ai/blog/glm-image模型https://modelscope.cn/models/ZhipuAI/GLM-Imagehttps://huggingface.co/zai-org/GLM-Image介绍GLM-Image 是一种图像生成模型采用混合自回归扩散解码器架构。在整体图像生成质量方面GLM-Image 与主流的潜在扩散方法相符但在文本渲染和知识密集型生成场景中展现出显著优势。它在需要精确语义理解和复杂信息表达的任务中表现尤为出色同时保持了高保真度和细粒度细节生成的强大能力。除了文本到图像的生成之外GLM-Image 还支持丰富的图像到图像任务包括图像编辑、风格迁移、身份保持生成和多主体一致性等。方法概述模型架构混合自回归扩散解码器设计。自回归生成器一个基于GLM-4-9B-0414初始化的 9B 参数模型并扩展了词汇表以包含视觉标记。该模型首先生成约 256 个标记的紧凑编码然后扩展到 1K 至 4K 个标记对应于 1K 至 2K 的高分辨率图像输出。扩散解码器一种基于单流 DiT 架构的 7B 参数解码器用于潜在空间图像解码。它配备了字形编码器文本模块显著提高了图像中文本的渲染精度。训练后采用解耦强化学习该模型引入了使用 GRPO 算法的细粒度模块化反馈策略从而显著提高了语义理解和视觉细节质量。自回归模块提供低频反馈信号侧重于美学和语义一致性从而提高指令遵循性和艺术表现力。解码器模块提供高频反馈以提高细节保真度和文本准确性从而实现高度逼真的纹理以及更精确的文本渲染。GLM-Image 支持在单个模型中同时生成文本到图像和图像到图像。文本转图像根据文本描述生成高细节图像在信息密集的场景中表现尤为出色。图像到图像支持多种任务包括图像编辑、风格迁移、多主体一致性以及人物和物体的身份保留生成。评估指标文本渲染基准测试长文本基准测试快速入门pip install githttps://github.com/huggingface/transformers.git pip install githttps://github.com/huggingface/diffusers.git文本转图像生成import torch from diffusers.pipelines.glm_image import GlmImagePipeline pipe GlmImagePipeline.from_pretrained(zai-org/GLM-Image, torch_dtypetorch.bfloat16, device_mapcuda) prompt A beautifully designed modern food magazine style dessert recipe illustration, themed around a raspberry mousse cake. The overall layout is clean and bright, divided into four main areas: the top left features a bold black title Raspberry Mousse Cake Recipe Guide, with a soft-lit close-up photo of the finished cake on the right, showcasing a light pink cake adorned with fresh raspberries and mint leaves; the bottom left contains an ingredient list section, titled Ingredients in a simple font, listing Flour 150g, Eggs 3, Sugar 120g, Raspberry puree 200g, Gelatin sheets 10g, Whipping cream 300ml, and Fresh raspberries, each accompanied by minimalist line icons (like a flour bag, eggs, sugar jar, etc.); the bottom right displays four equally sized step boxes, each containing high-definition macro photos and corresponding instructions, arranged from top to bottom as follows: Step 1 shows a whisk whipping white foam (with the instruction Whip egg whites to stiff peaks), Step 2 shows a red-and-white mixture being folded with a spatula (with the instruction Gently fold in the puree and batter), Step 3 shows pink liquid being poured into a round mold (with the instruction Pour into mold and chill for 4 hours), Step 4 shows the finished cake decorated with raspberries and mint leaves (with the instruction Decorate with raspberries and mint); a light brown information bar runs along the bottom edge, with icons on the left representing Preparation time: 30 minutes, Cooking time: 20 minutes, and Servings: 8. The overall color scheme is dominated by creamy white and light pink, with a subtle paper texture in the background, featuring compact and orderly text and image layout with clear information hierarchy. image pipe( promptprompt, height32 * 32, width36 * 32, num_inference_steps50, guidance_scale1.5, generatortorch.Generator(devicecuda).manual_seed(42), ).images[0] image.save(output_t2i.png)图生图import torch from diffusers.pipelines.glm_image import GlmImagePipeline from PIL import Image pipe GlmImagePipeline.from_pretrained(zai-org/GLM-Image, torch_dtypetorch.bfloat16, device_mapcuda) image_path cond.jpg prompt Replace the background of the snow forest with an underground station featuring an automatic escalator. image Image.open(image_path).convert(RGB) image pipe( promptprompt, image[image], # can input multiple images for multi-image-to-image generation such as [image, image1] height33 * 32, # Must set height even it is same as input image width32 * 32, # Must set width even it is same as input image num_inference_steps50, guidance_scale1.5, generatortorch.Generator(devicecuda).manual_seed(42), ).images[0] image.save(output_i2i.png)