The result? The local models missed important details or introduced hallucinations. So I switched to using cloud models, which produced better results but meant sacrificing privacy and offline capability.

Two weeks ago, Ollama released version 0.7.0 with improved support for vision models. They added support for three vision models I hadn't tested yet: Mistral 3.1, Qwen 2.5 VL and Gemma 3.

I decided to evaluate these models to see whether they've caught up to GPT-4 and Claude 3.5 in quality. Can local models now generate accurate and reliable alt-text?

Updating my alt-text script

For my earlier experiments, I created an open-source script that generates alt-text descriptions. The script is a Python wrapper around Simon Willison's llm tool, which provides a unified interface to LLMs. It supports models from Ollama, Hugging Face and various cloud providers.

To test the new models, I added 3 new entries to my script's models.yaml, which defines each model's prompt, temperature, and token settings. Once configured, generating alt-text is simple. Here is an example using the three new vision models:

$ ./caption.py test-images/image-1.jpg –model mistral-3.1-24b gemma3-27b qwen2.5vl-32b

Which outputs something like:

{
  "image": "test-images/image-1.jpg",
  "captions": {
    "mistral-3.1-24b": "A bustling intersection at night filled with pedestrians crossing in all directions."
    "gemma3-27b": "A high-angle view shows a crowded Tokyo street filled with pedestrians and brightly lit advertising billboards at night.",
    "qwen2.5vl-32b": "A bustling city intersection at night, crowded with people crossing the street, surrounded by tall buildings with bright, colorful billboards and advertisements.",
  }
}

Evaluating the models

To keep the results consistent, I used the same test images and the same evaluation method as in my earlier blog post. The details results are in this Google spreadsheet.

Each alt-text was scored from 0 to 5 based on three criteria: how well it identified the most important elements in the image, how effectively it captured the mood or atmosphere, and whether it avoided repetition, grammar issues or hallucinated details. I then converted each score into a letter grade from A to F.

For comparison, the cloud models received the following scores: GPT-4o earned an average of 4.8 out of 5 (grade A), and Claude 3.5 Sonnet received a perfect 5 out of 5 (grade A).

Test image 1: Shibuya Crossing, Tokyo

None earned an A because they failed to identify this as Shibuya Crossing in Tokyo, while cloud models do. Mistral also missed the billboards.

Test image 2: Isabella Stewart Gardner Museum, Boston

The vision models in my previous post often mistook the empty frame for a framed painting. All three models in this test correctly identified it as empty. Gemma and Qwen captured valuable details about the scene, while Mistral's description felt sparse.

Test image 3: wakeboarding in Vermont, USA

All three described a wakeboarding scene as "water skiing", while the cloud models correctly identified it as wakeboarding.

Test image 4: hiking in the Dolomites, Italy

Only Gemma recognized the statue as Mary. Both Mistral and Gemma missed the mountains in the background, which seems important.

Test image 5: backgammon by candlelight

Neither Mistral nor Gemma recognized the backgammon board. Only Qwen identified it correctly. Mistral also failed to capture the photo's mood.

Model accuracy

Qwen 2.5 VL performed best overall, with Gemma 3 not far behind.

Needless to say, these results are based on a small set of test images. And while I used a structured scoring system, the evaluation still involves subjective judgment. This is not a definitive ranking, but it's enough to draw some conclusions.

It was nice to say that all three LLMs avoided repetition and hallucinations, and generally captured the mood of the images.

Local models still make mistakes. All three described wakeboarding as "water skiing", most failed to recognize the statue as Mary or place the intersection in Japan. Cloud models get these details right, as I showed in my previous blog post.

Conclusion

I ran my original experiment four months ago, and at the time, none of the models I tested felt accurate enough for large-scale alt-text generation. Some, like Llama 3, showed promise but still fell short in overall quality.

Newer models like Qwen 2.5 VL and Gemma 3 have matched the performance I saw earlier with Llama 3. Both performed well in my latest test. They produced relevant, grounded descriptions without hallucinations or repetition, which earlier local models often struggled with.

Still, the quality is not yet at the level where I would trust these models to generate thousands of alt-texts without human review. They make more mistakes than GPT-4 or Claude 3.5.

My main question was: are local models now good enough for practical use? While Qwen 2.5 VL performed best overall, it still needs human review. I've started using it for small batches where manual checking is manageable. For large-scale, fully automated use, I continue using cloud models as they remain the most reliable option.

That said, local vision-language models continue to improve. My long-term goal is to return to a 100% local-first workflow that gives me more control and keeps my data private. While we're not there yet, these results show real progress.

My plan is to wait for the next generation of local vision models (or upgrade my hardware to run larger models). When those become available, I'll test them and report back.

What began as a simple task evolved into a four-part series where I compared different LLMs, evaluated local versus cloud processing, and built an automated workflow.

But this final step was different. It wasn't about technology. It was about trust and letting things go.

My AI tool in action

In my last blog post, I shared scripts to automate alt-text generation for a single image. The final step? Running my scripts on the 9,000 images missing alt-text. This covers over 20 years of images in photo albums and blog posts.

Here is my tool in action:

And yes, AI generated the alt-text for this GIF. AI describing AI, a recursion that should have ripped open the space-time continuum. Sadly, no portals appeared. At best, it might have triggered a stack overflow in a distant dimension. Meanwhile, I just did the evening dishes.

ChatGPT-4o processed all 9,000 images at half a cent each, for less than $50 in total. And despite hammering their service for a couple days, I never hit a rate limit or error. Very impressive.

AI is better than me

Trusting a script to label 9,000 images made me nervous. What if mistakes in auto-generated descriptions made my website less accessible? What if future AI models trained on any mistakes?

I started cautiously, stopping after each album to check every alt-text. After reviewing 250 images, I noticed something: I wasn't fixing errors, I was just tweaking words.

Then came the real surprise. I tested my script on albums I had manually described five years ago. The result was humbling. AI wrote better alt-text: spotting details I missed, describing scenes more clearly, and capturing nuances I overlooked. Turns out, past me wasn't so great at writing alt-text.

Not just that. The LLM understood Japanese restaurant menus, decoded Hungarian text, interpreted German Drupal books, and read Dutch street signs. It recognized conference badges and correctly labeled events. It understood cultural contexts across countries. It picked up details about my photos that I had forgotten or didn't even know existed.

I was starting to understand this wasn't about AI's ability to describe images; it was about me accepting that AI often described them better than I could.

Conclusion

AI isn't perfect, but it can be very useful. People worry about hallucinations and inaccuracy, and I did too. But after generating alt-text for 9,000 images, I saw something different: real, practical value.

It didn't just make my site more accessible; it challenged me. It showed me that sometimes, the best way to improve is to step aside and let a tool do the job better.

My own website is no exception, so a few weeks ago, I set out to add missing alt-text to about 9,000 images on this website.

What seemed like a simple fix became a multi-step challenge. I needed to evaluate different AI models and decide between local and cloud processing.

To make the web better, a lot of websites need to add alt-text to their images. So I decided to document my progress here on my blog so others can learn from it – or offer suggestions. This third post dives into the technical details of how I built an automated pipeline to generate alt-text at scale.

High-level architecture overview

My automation process follows three steps for each image:

1. Check if alt-text exists for a given image
2. Generate new alt-text using AI when missing
3. Update the database record for the image with the new alt-text

The rest of this post goes into more detail on each of these steps. If you're interested in the implementation, you can find most of the source code on GitHub.

Retrieving image metadata

To systematically process 9,000 images, I needed a structured way to identify which ones were missing alt-text.

Since my site runs on Drupal, I built two REST API endpoints to interact with the image metadata:

• GET /album/{album-name}/{image-name}/get – Retrieves metadata for an image, including title, alt-text, and caption.
• PATCH /album/{album-name}/{image-name}/patch – Updates specific fields, such as adding or modifying alt-text.

I've built similar APIs before, including one for my basement's temperature and humidity monitor. That post provides a more detailed breakdown of how I build endpoints like this.

This API uses separate URL paths (/get and /patch) for different operations, rather than using a single resource URL. I'd prefer to follow RESTful principles, but this approach avoids caching problems, including content negotiation issues in CDNs.

Anyway, with the new endpoints in place, fetching metadata for an image is simple:

curl -H "Authorization: test-token" \
  "https://dri.es/album/isle-of-skye-2024/journey-to-skye/get"

Every request requires an authorization token. And no, test-token isn't the real one. Without it, anyone could edit my images. While crowdsourced alt-text might be an interesting experiment, it's not one I'm looking to run today.

This request returns a JSON object with image metadata:

{
  "title": "Journey to Skye",
  "alt": "",
  "caption": "Each year, Klaas and I pick a new destination for our outdoor adventure. In 2024, we set off for the Isle of Skye in Scotland. This stop was near Glencoe, about halfway between Glasgow and Skye."
}

Because the alt-field is empty, the next step is to generate a description using AI.

Generating and refining alt-text with AI

In my first post on AI-generated alt-text, I wrote a Python script to compare 10 different local Large Language Models (LLMs). The script uses PyTorch, a widely used machine learning framework for AI research and deep learning. This implementation was a great learning experience.

The original script takes an image as input and generates alt-text using multiple LLMs:

./caption.py journey-to-skye.jpg
{
  "image": "journey-to-skye.jpg",
  "captions": {
    "vit-gpt2": "A man standing on top of a lush green field next to a body of water with a bird perched on top of it.",
    "git": "A man stands in a field next to a body of water with mountains in the background and a mountain in the background.",
    "blip": "This is an image of a person standing in the middle of a field next to a body of water with a mountain in the background.",
    "blip2-opt": "A man standing in the middle of a field with mountains in the background.",
    "blip2-flan": "A man is standing in the middle of a field with a river and mountains behind him on a cloudy day.",
    "minicpm-v": "A person standing alone amidst nature, with mountains and cloudy skies as backdrop.",
    "llava-13b": "A person standing alone in a misty, overgrown field with heather and trees, possibly during autumn or early spring due to the presence of red berries on the trees and the foggy atmosphere.",
    "llava-34b": "A person standing alone on a grassy hillside with a body of water and mountains in the background, under a cloudy sky.",
    "llama32-vision-11b": "A person standing in a field with mountains and water in the background, surrounded by overgrown grass and trees."
  }
}

My original plan was to run everything locally for full control, no subscription costs, and optimal privacy. But after testing 10 local LLMs, I changed my mind.

I knew cloud-based models would be better, but wanted to see if local models were good enough for alt-texts. Turns out, they're not quite there. You can read the full comparison, but I gave the best local models a B, while cloud models earned an A.

While local processing aligned with my principles, it compromised the primary goal: creating the best possible descriptions for screen reader users. So I abandoned my local-only approach and decided to use cloud-based LLMs.

To automate alt-text generation for 9,000 images, I needed programmatic access to cloud models rather than relying on their browser-based interfaces – though browser-based AI can be tons of fun.

Instead of expanding my script with cloud LLM support, I switched to Simon Willison's llm tool: https://llm.datasette.io/. llm is a command-line tool and Python library that supports both local and cloud-based models. It takes care of installation, dependencies, API key management, and uploading images. Basically, all the things I didn't want to spend time maintaining myself.

Despite enjoying my PyTorch explorations with vision-language models and multimodal encoders, I needed to focus on results. My weekly progress goal meant prioritizing working alt-text over building homegrown inference pipelines.

I also considered you, my readers. If this project inspires you to make your own website more accessible, you're better off with a script built on a well-maintained tool like llm rather than trying to adapt my custom implementation.

Scrapping my PyTorch implementation stung at first, but building on a more mature and active open-source project was far better for me and for you. So I rewrote my script, now in the v2 branch, with the original PyTorch version preserved in v1.

The new version of my script keeps the same simple interface but now supports cloud models like ChatGPT and Claude:

./caption.py journey-to-skye.jpg --model chatgpt-4o-latest claude-3-sonnet --context "Location: Glencoe, Scotland"
{
  "image": "journey-to-skye.jpg",
  "captions": {
    "chatgpt-4o-latest": "A person in a red jacket stands near a small body of water, looking at distant mountains in Glencoe, Scotland.",
    "claude-3-sonnet": "A person stands by a small lake surrounded by grassy hills and mountains under a cloudy sky in the Scottish Highlands."
  }
}

The --context parameter improves alt-text quality by adding details the LLM can't determine from the image alone. This might include GPS coordinates, album titles, or even a blog post about the trip.

In this example, I added "Location: Glencoe, Scotland". Notice how ChatGPT-4o mentions Glencoe directly while Claude-3 Sonnet references the Scottish Highlands. This contextual information makes descriptions more accurate and valuable for users. For maximum accuracy, use all available information!

Updating image metadata

With alt-text generated, the final step is updating each image. The PATCH endpoint accepts only the fields that need changing, preserving other metadata:

curl -X PATCH \
  -H "Authorization: test-token" \
  "https://dri.es/album/isle-of-skye-2024/journey-to-skye/patch" \
  -d '{
    "alt": "A person stands by a small lake surrounded by grassy hills and mountains under a cloudy sky in the Scottish Highlands.",
  }'

That's it. This completes the automation loop for one image. It checks if alt-text is needed, creates a description using a cloud-based LLM, and updates the image if necessary. Now, I just need to do this about 9,000 times.

Tracking AI-generated alt-text

Before running the script on all 9,000 images, I added a label to the database that marks each alt-text as either human-written or AI-generated. This makes it easy to:

• Re-run AI-generated descriptions without overwriting human-written ones
• Upgrade AI-generated alt-text as better models become available

With this approach I can update the AI-generated alt-text when a future ChatGPT model is released. And eventually, it might allow me to return to my original principles: to use a high-quality local LLM trained on public domain data. In the meantime, it helps me make the web more accessible today while building toward a better long-term solution tomorrow.

Next steps

Now that the process is automated for a single image, the last step is to run the script on all 9,000. And honestly, it makes me nervous. The perfectionist in me wants to review every single AI-generated alt-text, but that is just not feasible. So I have to trust AI. I'll probably write one more post to share the results and what I learned from this final step.

Stay tuned.

1. Should I use AI-generated alt-texts, even if it they are not perfect?
2. Should I generate these alt-texts with local LLMs or in the cloud?

Since then, I've received dozens of emails and LinkedIn comments. The responses were all over the place. Some swore by local models because they align with open-source values. Others championed cloud-based LLMs for better accuracy. A couple of people even ran tests using different models to help me out.

I appreciate every response. It's a great reminder of why building in the open is so valuable: it brings in diverse perspectives.

But one comment stood out. A visually impaired reader put it simply: Imperfect alt-text is better than no alt-text.

That comment made the first decision easy: AI-generated alt-text, even if not perfect, is better than nothing.

The harder question was which AI models to use. As a long-term open-source evangelist, I really want to run my own LLMs. Local AI aligns with my values: no privacy concerns, no API quotas, more transparency, and more control. They also align with my wallet: no subscription fees. And, let's be honest: running your own LLMs earns you some bragging rights at family parties.

But here is the problem: local models aren't as good as cloud models.

Most laptops and consumer desktops have 16–32GB of RAM, which limits them to small, lower-accuracy models. Even maxing out an Apple Mac Studio with 192GB of RAM doesn't change that. Gaming GPUs? Also a dead end, at least for me. Even high-end cards with 24GB of VRAM struggle with the larger models unless you stack multiple cards together.

The gap between local and cloud hardware is big. It's like racing a bicycle against a jet engine.

I could wait. Apple will likely release a new Mac Studio this year, and I'm hoping it supports more than 192GB of RAM. NVIDIA's Digits project could make consumer-grade LLM hardware even more viable.

Local models are also improving fast. Just in the past few weeks:

• Alibaba released Qwen 2.5 VL, which performs well in benchmarks.
• DeepSeek launched DeepSeek-VL2, a strong new open model.
• Mark Zuckerberg shared that Meta's Llama 4 is in testing and might be released in the next few months.

Consumer hardware and local models will continue to improve. But even when they do, cloud models will still be ahead. So, I am left with this choice:

1. Prioritize accessibility: use the best AI models available today, even if they're cloud-based.
2. Stick to Open Source ideals: run everything locally, but accept worse accuracy.

A reader, Kris, put it well: Prioritize users while investing in your values. That stuck with me.

I'd love to run everything locally, but making my content accessible and ensuring its accuracy matters more. So, for now, I'm moving forward with cloud-based models, even if it means compromising on my open-source ideals.

It's not the perfect answer, but it's the practical one. Prioritizing accessibility and end-user needs over my own principles feels like the right choice.

That doesn't mean I'm giving up on local LLMs. I'll keep testing models, tracking improvements, and looking for the right hardware upgrades. The moment local AI is good enough for generating alt-text, I'll switch. In my next post, I'll share my technical approach to making this work.

When I started my blog nearly 20 years ago, I didn't think much about alt-texts. Over time, I realized its importance for visually impaired users who rely on screen readers.

The past 5+ years, I diligently added alt-text to every new image I uploaded. But that only covers about 1,000 images, leaving most older photos without descriptions.

Writing 9,000 alt-texts manually would take ages. Of course, AI could do this much faster, but is it good enough?

To see what AI can do, I tested 12 Large Language Models (LLMs): 10 running locally and 2 in the cloud. My goal was to test their accuracy and determine whether they can generate accurate alt-text.

The TL;DR is that, not surprisingly, cloud models (GPT-4, Claude Sonnet 3.5) set the benchmark with A-grade performance, though not 100% perfect. I prefer local models for privacy, cost, and offline use. Among local options, the Llama variants and MiniCPM-V perform best. Both earned a B grade: they work reliably but sometimes miss important details.

I know I'm not the only one. Plenty of people – entire organizations even – have massive backlogs of images without alt-text. I'm determined to fix that for my blog and share what I learn along the way. This blog post is just step one – subscribe by email or RSS to get future posts.

Models evaluated

I tested alt-text generation using 12 AI models: 9 on my MacBook Pro with 32GB RAM, 1 on a higher-RAM machine (thanks to Jeremy Andrews, a friend and long-time Drupal contributor), and 2 cloud-based services.

The table below lists the models I tested, with details like links to research papers, release dates, parameter sizes (in billions), memory requirements, some architectural details and more:

How image-to-text models work (in less than 30 seconds)

LLMs come in many forms, but for this project, I focused on image-to-text and multi-modal models. Both types of models can analyze images and generate text, either by describing images or answering questions about them.

Image-to-text models follow a two-step process: vision encoding and language decoding:

1. Vision encoding: First, the model breaks an image down into patches. You can think of these as "puzzle pieces". The patches are converted into mathematical representations called embeddings, which summarize their visual details. Next, an attention mechanism filters out the most important patches (e.g. the puzzle pieces with the cat's outline or fur texture) and eliminates less relevant details (e.g. puzzle pieces with plain blue skies).
2. Language encoding: Once the model has summarized the most important visual features, it uses a language model to translate those features into words. This step is where the actual text (image captions or Q&A answers) is generated.

In short, the vision encoder sees the image, while the language encoder describes it.

If you look at the table above, you'll see that each row pairs a vision encoder (e.g., ViT, CLIP, Swin) with a language encoder (e.g., GPT-2, BERT, T5, Llama).

For a more in-depth explanation, I recommend Sebastian Raschka's article Understanding Multi-modal LLMs, which also covers how image encoders work. It's fantastic!

Comparing different AI models

I wrote a Python script that generates alt-texts for images using nine different local models. You can find it in my GitHub repository. It takes care of installing models, running them, and generating alt-texts. It supports both Hugging Face and Ollama and is built to be easily extended as new models come out.

You can run the script as follows:

$ ./caption.py ./test-images/image-1.jpg

The first time you run the script, it will download all models, which requires significant disk space and bandwidth – expect to download over 50GB of model data.

The script outputs a JSON response, making it easy to integrate or analyze programmatically. Here is an example output:

  {
  "image": "test-images/image-1.jpg",
  "alt-texts": {
  "vit-gpt2": "A city at night with skyscrapers and a traffic light on the side of the street in front of a tall building.",
  "git": "A busy city street is lit up at night, with the word qroi on the right side of the sign.",
  "blip": "This is an aerial view of a busy city street at night with lots of people walking and cars on the side of the road.",
  "blip2-opt": "An aerial view of a busy city street at night.",
  "blip2-flan": "An aerial view of a busy street in tokyo, japanese city at night with large billboards.",
  "minicpm-v": "A bustling cityscape at night with illuminated billboards and advertisements, including one for Michael Kors.",
  "llava-13b": "A bustling nighttime scene from Tokyo's famous Shibuya Crossing, characterized by its bright lights and dense crowds of people moving through the intersection.",
  "llava-34b": "A bustling city street at night, filled with illuminated buildings and numerous pedestrians.",
  "llama32-vision-11b": "A bustling city street at night, with towering skyscrapers and neon lights illuminating the scene."
  }
  }

Test images

With the script ready, I decided to test it on some of my 10,000 photos. Not all of them at once. I picked five that I consider non-standard. Instead of simple portraits or landscapes, I picked photos with elements that might confuse or challenge the models.

One photo is from the Isabella Stewart Gardner Museum in Boston and features an empty gold frame. The frame once held a masterpiece stolen in the infamous 1990 heist, one of the biggest art thefts in history. I wanted to see if the models would recognize it as empty or mistake it for a framed painting.

Another photo, taken last summer in Vermont, shows a wakeboarder. Though he is the main subject, he is relatively small in the frame. I was curious to see if the models could still recognize him as the focal point.

In another photo, a backgammon game is set in a dark but cozy atmosphere. I was curious to see if the models could recognize partially visible objects and capture the mood of the scene.

To ensure a fair test, I stripped all EXIF metadata from the images. This includes any embedded captions, GPS coordinates, or other details that could inadvertently help the models.

Yes, I know that a test set of five images is small, but it's sufficient to identify the top models for further evaluation. With 12 models generating alt-texts for each photo, I had to manually evaluate 60 alt-texts, which was tedious to do fairly and accurately. For now, these five images are enough to filter out weaker models and pinpoint the best 2-3 LLMs for more detailed testing.

Next, I'll share the detailed results for each of the five photos. It's a lot to go through, so feel free to skip ahead to the summary.

Test image 1: Shibuya Crossing, Tokyo

Benchmark LLMs:

Local LLMs:

Test image 2: Isabella Stewart Gardner Museum, Boston

Benchmark LLMs:

Local LLMs:

Test image 3: wakeboarding in Vermont, USA

Benchmark LLMs:

Local LLMs:

Test image 4: hiking in the Dolomites, Italy

Benchmark LLMs:

Local LLMs:

Test image 5: backgammon by candlelight

Benchmark LLMs:

Local LLMs:

Model accuracy

I evaluated each description using a structured but subjective scoring system. For each image, I identified the two or three most important objects the AI should recognize and include in its description. I also assessed whether the model captured the photo's mood, which can be important for visually impaired users. Finally, I deducted points for repetition, grammar errors, or hallucinations (invented details). Each alt-text received a score from 0 to 5, which I then converted to a letter grade from A to F.

The cloud-based models, GPT-4o and Claude 3.5 Sonnet, performed nearly perfectly on my small test of five images, with no major errors, hallucinations, repetitions and excellent mood detection.

Among local models, both Llama variants and MiniCPM-V show the strongest performance.

Repetition in descriptions frustrates users of screen readers. Early models like VIT-GPT2, GIT, BLIP, and BLIP2 frequently repeat content, making them unsuitable.

Hallucinations can be a serious issue in my opinion. Describing nonexistent objects or actions misleads visually impaired users and erodes trust. Among the best-performing local models, MiniCPM-V did not hallucinate, while Llama 11B and Llama 90B each made one mistake. Llama 90B misidentified a cabinet at the museum as a table, and Llama 11B described multiple people wakeboarding instead of just one. While these errors aren't dramatic, they are still frustrating.

Capturing mood is essential for giving visually impaired users a richer understanding of images. While early models struggled in this area, all recent models all performed well. This includes both LLaVA variants and MiniCPM-V.

From a practical standpoint, Llama 11B and MiniCPM-V ran smoothly on my 32GB RAM laptop, but Llama 90B needed more memory. Long story short, this means that Llama 11B and MiniCPM-V are my best candidates for additional testing.

Possible next steps

The results raise a tough question: is a "B"-level alt-text better than none at all? Many human-written alt-texts probably aren't perfect either. Should I wait for local models to hit an "A"-grade, or is an imperfect description still better than no alt-text at all?

Here are four possible next steps:

1. Combine AI outputs – Run the same image through different models and merge their results to try and create more accurate descriptions.
2. Wait and upgrade – Use the best local model for now, tag AI-generated alt-texts in the database, and refresh them in 6–12 months when new and better local models are available.
3. Go cloud-based – Get the best quality with a cloud model, even if it means uploading 65GB of photos. I can't explain why, or if the feeling is even justified, but it feels like giving in.
4. Hybrid approach – Use AI to generate alt-texts but review them manually. With 9,000 images, that is not practical. I'd need a way to flag alt-texts most likely to be wrong. Can LLMs give me a reliably confidence score?

Each option comes with trade-offs. Some options are quick but imperfect, others take work but might be worth it. Going cloud-based is the easiest but it feels like giving in. Waiting for better models is effortless but means delaying progress. Merging AI outputs or assigning a confidence score takes more effort but might be the best balance of speed and accuracy.

Maybe the solution is a combination of these options? I could go cloud-based now, tag the AI-generated alt-texts in my database, and regenerate them in 6–12 months when LLMs got even better.

It also comes down to pragmatism versus principle. Should I stick to local models because I believe in data privacy and Open Source, or should I prioritize accessibility by providing the best possible alt-text for users? The local-first approach better aligns with my values, but it might come at the cost of a worse experience for visually impaired users.

I'll be weighing these options over the next few weeks. What would you do? I'd love to hear your thoughts!

Update: My thoughts on using AI for alt-text has evolved across several blog posts. First, I chose a cloud-based LLM after all. Then, I built an automated system to generate and update descriptions for just one image. Finally, I scaled it to 9,000 images and learned to trust AI in the process.