Flux by Black Forest Labs: The Next Leap in Text-to-Image Models. Is it better than Midjourney?

Black Forest Labs, the team behind the groundbreaking Stable Diffusion model, has released Flux – a suite of state-of-the-art models that promise to redefine the capabilities of AI-generated imagery. But does Flux truly represent a leap forward in the field, and how does it stack up against industry leaders like Midjourney? Let’s dive deep into the world of Flux and explore its potential to reshape the future of AI-generated art and media.

The Birth of Black Forest Labs

Before we delve into the technical aspects of Flux, it’s crucial to understand the pedigree behind this innovative model. Black Forest Labs is not just another AI startup; it’s a powerhouse of talent with a track record of developing foundational generative AI models. The team includes the creators of VQGAN, Latent Diffusion, and the Stable Diffusion family of models that have taken the AI art world by storm.

With a successful Series Seed funding round of $31 million led by Andreessen Horowitz and support from notable angel investors, Black Forest Labs has positioned itself at the forefront of generative AI research. Their mission is clear: to develop and advance state-of-the-art generative deep learning models for media such as images and videos, while pushing the boundaries of creativity, efficiency, and diversity.

Introducing the Flux Model Family

Black Forest Labs has introduced the FLUX.1 suite of text-to-image models, designed to set new benchmarks in image detail, prompt adherence, style diversity, and scene complexity. The Flux family consists of three variants, each tailored to different use cases and accessibility levels:

1. FLUX.1 [pro]: The flagship model, offering top-tier performance in image generation with superior prompt following, visual quality, image detail, and output diversity. Available through an API, it’s positioned as the premium option for professional and enterprise use.
2. FLUX.1 [dev]: An open-weight, guidance-distilled model for non-commercial applications. It’s designed to achieve similar quality and prompt adherence capabilities as the pro version while being more efficient.
3. FLUX.1 [schnell]: The fastest model in the suite, optimized for local development and personal use. It’s openly available under an Apache 2.0 license, making it accessible for a wide range of applications and experiments.

I’ll provide some unique and creative prompt examples that showcase FLUX.1’s capabilities. These prompts will highlight the model’s strengths in handling text, complex compositions, and challenging elements like hands.

• Artistic Style Blending with Text: “Create a portrait of Vincent van Gogh in his signature style, but replace his beard with swirling brush strokes that form the words ‘Starry Night’ in cursive.”

Black Forest Labs Open-Source FLUX.1

• Dynamic Action Scene with Text Integration: “A superhero bursting through a comic book page. The action lines and sound effects should form the hero’s name ‘FLUX FORCE’ in bold, dynamic typography.”

Black Forest Labs Open-Source FLUX.1

• Surreal Concept with Precise Object Placement: “Close-up of a cute cat with brown and white colors under window sunlight. Sharp focus on eye texture and color. Natural lighting to capture authentic eye shine and depth.”

Black Forest Labs Open-Source FLUX.1

These prompts are designed to challenge FLUX.1’s capabilities in text rendering, complex scene composition, and detailed object creation, while also showcasing its potential for creative and unique image generation.

Technical Innovations Behind Flux

At the heart of Flux’s impressive capabilities lies a series of technical innovations that set it apart from its predecessors and contemporaries:

Transformer-powered Flow Models at Scale

All public FLUX.1 models are built on a hybrid architecture that combines multimodal and parallel diffusion transformer blocks, scaled to an impressive 12 billion parameters. This represents a significant leap in model size and complexity compared to many existing text-to-image models.

The Flux models improve upon previous state-of-the-art diffusion models by incorporating flow matching, a general and conceptually simple method for training generative models. Flow matching provides a more flexible framework for generative modeling, with diffusion models being a special case within this broader approach.

To enhance model performance and hardware efficiency, Black Forest Labs has integrated rotary positional embeddings and parallel attention layers. These techniques allow for better handling of spatial relationships in images and more efficient processing of large-scale data.

Architectural Innovations

Let’s break down some of the key architectural elements that contribute to Flux’s performance:

1. Hybrid Architecture: By combining multimodal and parallel diffusion transformer blocks, Flux can effectively process both textual and visual information, leading to better alignment between prompts and generated images.
2. Flow Matching: This approach allows for more flexible and efficient training of generative models. It provides a unified framework that encompasses diffusion models and other generative techniques, potentially leading to more robust and versatile image generation.
3. Rotary Positional Embeddings: These embeddings help the model better understand and maintain spatial relationships within images, which is crucial for generating coherent and detailed visual content.
4. Parallel Attention Layers: This technique allows for more efficient processing of attention mechanisms, which are critical for understanding relationships between different elements in both text prompts and generated images.
5. Scaling to 12B Parameters: The sheer size of the model allows it to capture and synthesize more complex patterns and relationships, potentially leading to higher quality and more diverse outputs.

Benchmarking Flux: A New Standard in Image Synthesis

https://blackforestlabs.ai/announcing-black-forest-labs/

Black Forest Labs claims that FLUX.1 sets new standards in image synthesis, surpassing popular models like Midjourney v6.0, DALL·E 3 (HD), and SD3-Ultra in several key aspects:

1. Visual Quality: Flux aims to produce images with higher fidelity, more realistic details, and better overall aesthetic appeal.
2. Prompt Following: The model is designed to adhere more closely to the given text prompts, generating images that more accurately reflect the user’s intentions.
3. Size/Aspect Variability: Flux supports a diverse range of aspect ratios and resolutions, from 0.1 to 2.0 megapixels, offering flexibility for various use cases.
4. Typography: The model shows improved capabilities in generating and rendering text within images, a common challenge for many text-to-image models.
5. Output Diversity: Flux is specifically fine-tuned to preserve the entire output diversity from pretraining, offering a wider range of creative possibilities.

Flux vs. Midjourney: A Comparative Analysis

https://blackforestlabs.ai/announcing-black-forest-labs/

Now, let’s address the burning question: Is Flux better than Midjourney? To answer this, we need to consider several factors:

Image Quality and Aesthetics

Both Flux and Midjourney are known for producing high-quality, visually stunning images. Midjourney has been praised for its artistic flair and ability to create images with a distinct aesthetic appeal. Flux, with its advanced architecture and larger parameter count, aims to match or exceed this level of quality.

Early examples from Flux show impressive detail, realistic textures, and a strong grasp of lighting and composition. However, the subjective nature of art makes it difficult to definitively claim superiority in this area. Users may find that each model has its strengths in different styles or types of imagery.

Prompt Adherence

One area where Flux potentially edges out Midjourney is in prompt adherence. Black Forest Labs has emphasized their focus on improving the model’s ability to accurately interpret and execute on given prompts. This could result in generated images that more closely match the user’s intentions, especially for complex or nuanced requests.

Midjourney has sometimes been criticized for taking creative liberties with prompts, which can lead to beautiful but unexpected results. Flux’s approach may offer more precise control over the generated output.

Speed and Efficiency

With the introduction of FLUX.1 [schnell], Black Forest Labs is targeting one of Midjourney’s key advantages: speed. Midjourney is known for its rapid generation times, which has made it popular for iterative creative processes. If Flux can match or exceed this speed while maintaining quality, it could be a significant selling point.

Accessibility and Ease of Use

Midjourney has gained popularity partly due to its user-friendly interface and integration with Discord. Flux, being newer, may need time to develop similarly accessible interfaces. However, the open-source nature of FLUX.1 [schnell] and [dev] models could lead to a wide range of community-developed tools and integrations, potentially surpassing Midjourney in terms of flexibility and customization options.

Technical Capabilities

Flux’s advanced architecture and larger model size suggest that it may have more raw capability in terms of understanding complex prompts and generating intricate details. The flow matching approach and hybrid architecture could allow Flux to handle a wider range of tasks and generate more diverse outputs.

Ethical Considerations and Bias Mitigation

Both Flux and Midjourney face the challenge of addressing ethical concerns in AI-generated imagery, such as bias, misinformation, and copyright issues. Black Forest Labs’ emphasis on transparency and their commitment to making models widely accessible could potentially lead to more robust community oversight and faster improvements in these areas.

Code Implementation and Deployment

Using Flux with Diffusers

Flux models can be easily integrated into existing workflows using the Hugging Face Diffusers library. Here’s a step-by-step guide to using FLUX.1 [dev] or FLUX.1 [schnell] with Diffusers:

1. First, install or upgrade the Diffusers library:

!pip install git+https://github.com/huggingface/diffusers.git

2. Then, you can use the FluxPipeline to run the model:

import torch
from diffusers import FluxPipeline
# Load the model
pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
# Enable CPU offloading to save VRAM (optional)
pipe.enable_model_cpu_offload()
# Generate an image
prompt = "A cat holding a sign that says hello world"
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    output_type="pil",
    num_inference_steps=50,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]
# Save the generated image
image.save("flux-dev.png")

Deploying Flux as an API with LitServe

For those looking to deploy Flux as a scalable API service, Black Forest Labs provides an example using LitServe, a high-performance inference engine. Here’s a breakdown of the deployment process:

Define the model server:

from io import BytesIO
from fastapi import Response
import torch
import time
import litserve as ls
from optimum.quanto import freeze, qfloat8, quantize
from diffusers import FlowMatchEulerDiscreteScheduler, AutoencoderKL
from diffusers.models.transformers.transformer_flux import FluxTransformer2DModel
from diffusers.pipelines.flux.pipeline_flux import FluxPipeline
from transformers import CLIPTextModel, CLIPTokenizer, T5EncoderModel, T5TokenizerFast
class FluxLitAPI(ls.LitAPI):
    def setup(self, device):
        # Load model components
        scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained("black-forest-labs/FLUX.1-schnell", subfolder="scheduler")
        text_encoder = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14", torch_dtype=torch.bfloat16)
        tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14", torch_dtype=torch.bfloat16)
        text_encoder_2 = T5EncoderModel.from_pretrained("black-forest-labs/FLUX.1-schnell", subfolder="text_encoder_2", torch_dtype=torch.bfloat16)
        tokenizer_2 = T5TokenizerFast.from_pretrained("black-forest-labs/FLUX.1-schnell", subfolder="tokenizer_2", torch_dtype=torch.bfloat16)
        vae = AutoencoderKL.from_pretrained("black-forest-labs/FLUX.1-schnell", subfolder="vae", torch_dtype=torch.bfloat16)
        transformer = FluxTransformer2DModel.from_pretrained("black-forest-labs/FLUX.1-schnell", subfolder="transformer", torch_dtype=torch.bfloat16)
        # Quantize to 8-bit to fit on an L4 GPU
        quantize(transformer, weights=qfloat8)
        freeze(transformer)
        quantize(text_encoder_2, weights=qfloat8)
        freeze(text_encoder_2)
        # Initialize the Flux pipeline
        self.pipe = FluxPipeline(
            scheduler=scheduler,
            text_encoder=text_encoder,
            tokenizer=tokenizer,
            text_encoder_2=None,
            tokenizer_2=tokenizer_2,
            vae=vae,
            transformer=None,
        )
        self.pipe.text_encoder_2 = text_encoder_2
        self.pipe.transformer = transformer
        self.pipe.enable_model_cpu_offload()
    def decode_request(self, request):
        return request["prompt"]
    def predict(self, prompt):
        image = self.pipe(
            prompt=prompt, 
            width=1024,
            height=1024,
            num_inference_steps=4, 
            generator=torch.Generator().manual_seed(int(time.time())),
            guidance_scale=3.5,
        ).images[0]
        return image
    def encode_response(self, image):
        buffered = BytesIO()
        image.save(buffered, format="PNG")
        return Response(content=buffered.getvalue(), headers={"Content-Type": "image/png"})
# Start the server
if __name__ == "__main__":
    api = FluxLitAPI()
    server = ls.LitServer(api, timeout=False)
    server.run(port=8000)

This code sets up a LitServe API for Flux, including model loading, request handling, image generation, and response encoding.

Start the server:

</pre>
python server.py
<pre>

Use the model API:

You can test the API using a simple client script:

import requests
import json
url = "http://localhost:8000/predict"
prompt = "a robot sitting in a chair painting a picture on an easel of a futuristic cityscape, pop art"
response = requests.post(url, json={"prompt": prompt})
with open("generated_image.png", "wb") as f:
    f.write(response.content)
print("Image generated and saved as generated_image.png")

Key Features of the Deployment

1. Serverless Architecture: The LitServe setup allows for scalable, serverless deployment that can scale to zero when not in use.
2. Private API: You can deploy Flux as a private API on your own infrastructure.
3. Multi-GPU Support: The setup is designed to work efficiently across multiple GPUs.
4. Quantization: The code demonstrates how to quantize the model to 8-bit precision, allowing it to run on less powerful hardware like NVIDIA L4 GPUs.
5. CPU Offloading: The enable_model_cpu_offload() method is used to conserve GPU memory by offloading parts of the model to CPU when not in use.

Practical Applications of Flux

The versatility and power of Flux open up a wide range of potential applications across various industries:

1. Creative Industries: Graphic designers, illustrators, and artists can use Flux to quickly generate concept art, mood boards, and visual inspirations.
2. Marketing and Advertising: Marketers can create custom visuals for campaigns, social media content, and product mockups with unprecedented speed and quality.
3. Game Development: Game designers can use Flux to rapidly prototype environments, characters, and assets, streamlining the pre-production process.
4. Architecture and Interior Design: Architects and designers can generate realistic visualizations of spaces and structures based on textual descriptions.
5. Education: Educators can create custom visual aids and illustrations to enhance learning materials and make complex concepts more accessible.
6. Film and Animation: Storyboard artists and animators can use Flux to quickly visualize scenes and characters, accelerating the pre-visualization process.

The Future of Flux and Text-to-Image Generation

Black Forest Labs has made it clear that Flux is just the beginning of their ambitions in the generative AI space. They’ve announced plans to develop competitive generative text-to-video systems, promising precise creation and editing capabilities at high definition and unprecedented speed.

This roadmap suggests that Flux is not just a standalone product but part of a broader ecosystem of generative AI tools. As the technology evolves, we can expect to see:

1. Improved Integration: Seamless workflows between text-to-image and text-to-video generation, allowing for more complex and dynamic content creation.
2. Enhanced Customization: More fine-grained control over generated content, possibly through advanced prompt engineering techniques or intuitive user interfaces.
3. Real-time Generation: As models like FLUX.1 [schnell] continue to improve, we may see real-time image generation capabilities that could revolutionize live content creation and interactive media.
4. Cross-modal Generation: The ability to generate and manipulate content across multiple modalities (text, image, video, audio) in a cohesive and integrated manner.
5. Ethical AI Development: Continued focus on developing AI models that are not only powerful but also responsible and ethically sound.

Conclusion: Is Flux Better Than Midjourney?

The question of whether Flux is “better” than Midjourney is not easily answered with a simple yes or no. Both models represent the cutting edge of text-to-image generation technology, each with its own strengths and unique characteristics.

Flux, with its advanced architecture and emphasis on prompt adherence, may offer more precise control and potentially higher quality in certain scenarios. Its open-source variants also provide opportunities for customization and integration that could be highly valuable for developers and researchers.

Midjourney, on the other hand, has a proven track record, a large and active user base, and a distinctive artistic style that many users have come to love. Its integration with Discord and user-friendly interface have made it highly accessible to creatives of all technical skill levels.

Ultimately, the “better” model may depend on the specific use case, personal preferences, and the evolving capabilities of each platform. What’s clear is that Flux represents a significant step forward in the field of generative AI, introducing innovative techniques and pushing the boundaries of what’s possible in text-to-image synthesis.