First complete test run!

[dbuytaert]

My script has reached a point where I'm ready to document the first complete run.

The captions.json file below was generated using the following command:

find . -name "*.jpg" -exec ./caption {} --time \; | jq -s '{"results": .}' > captions.json

It generates captions for all test images using all available models, and combines the results into a single JSON file. (You need to have jq installed for this.)

I plan to share a more detailed write-up and analysis in a future blog post on https://dri.es/. Until then, I might tinker with this script some more and welcome any suggestions or improvements.

Happy holidays! 🎄

{
  "results": [
    {
      "image": "./test-images/image-9.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A bird perched on top of a wooden post in front of a mountain range with a view of a mountain range and mountains.",
          "time": 10
        },
        "git": {
          "caption": "A wooden sign in front of a mountain with a statue on it that says \" meise museum \" on the top.",
          "time": 21
        },
        "blip": {
          "caption": "There is a birdhouse with a statue on top of it in the middle of a field with trees and mountains in the background.",
          "time": 7
        },
        "blip2-opt": {
          "caption": "A statue of mary on a wooden post in the middle of a field with mountains in the background.",
          "time": 34
        },
        "blip2-flan": {
          "caption": "A statue of the virgin mary sits on top of a wooden post in the middle of a mountain.",
          "time": 39
        },
        "llama32-vision-11b": {
          "caption": "A wooden shrine with a statue of mary and a signpost in front of mountains.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-8.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A man in a bathing suit is sitting on a boogie board at the water's edge with his feet propped up.",
          "time": 11
        },
        "git": {
          "caption": "A person is standing in the water and one of them is wearing a pair of shorts with the words \" i love you \" on it.",
          "time": 26
        },
        "blip": {
          "caption": "There is a man that is standing on the edge of a swimming pool with his feet in the water and a hose attached to the side of the pool.",
          "time": 7
        },
        "blip2-opt": {
          "caption": "A man in swim trunks standing on the edge of a pool.",
          "time": 26
        },
        "blip2-flan": {
          "caption": "A young boy is leaning on the railing of a dock in a body of water with his feet in the water.",
          "time": 40
        },
        "llama32-vision-11b": {
          "caption": "A person standing on a dock, with their legs submerged in water and holding onto a metal archway.",
          "time": 46
        }
      }
    },
    {
      "image": "./test-images/image-10.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A city at night with skyscrapers and a traffic light on the side of the street in front of a tall building.",
          "time": 12
        },
        "git": {
          "caption": "A busy city street is lit up at night, with the word qroi on the right side of the sign.",
          "time": 22
        },
        "blip": {
          "caption": "This is an aerial view of a busy city street at night with lots of people walking and cars on the side of the road.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "An aerial view of a busy city street at night.",
          "time": 32
        },
        "blip2-flan": {
          "caption": "An aerial view of a busy street in tokyo, japanese city at night with large billboards.",
          "time": 41
        },
        "llama32-vision-11b": {
          "caption": "A bustling city street at night, with numerous billboards and advertisements lining the buildings.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-11.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A building that has a lot of windows on top of it and some cars parked on the side of the road in front of it.",
          "time": 10
        },
        "git": {
          "caption": "A laptop sits on a table in front of a cityscape, with the skyline in the background and a cityscape in the background.",
          "time": 24
        },
        "blip": {
          "caption": "There is a laptop sitting on top of a table on a balcony with a cityscape in the background and a solar panel on top of it.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A solar panel on top of a table in front of a city skyline.",
          "time": 27
        },
        "blip2-flan": {
          "caption": "A solar panel sits on top of a railing on a balcony overlooking a city skyline.",
          "time": 40
        },
        "llama32-vision-11b": {
          "caption": "A solar panel on top of a table with a city skyline in the background.",
          "time": 45
        }
      }
    },
    {
      "image": "./test-images/image-12.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A man standing on top of a boat next to another man holding a surfboard in one hand and a surfboard in the other.",
          "time": 11
        },
        "git": {
          "caption": "Two men are in a boat, one of them is wearing an orange hat and the other is wearing an orange hat.",
          "time": 21
        },
        "blip": {
          "caption": "There are two men riding on the back of a boat in the water, one of them is on a surfboard and the other is on a board.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "Three young men sitting on the back of a boat.",
          "time": 26
        },
        "blip2-flan": {
          "caption": "A group of people sitting on a boat watching a man ride a surfboard in the middle of the water.",
          "time": 38
        },
        "llama32-vision-11b": {
          "caption": "Two shirtless men on a boat watching people wakeboarding or surfing behind it.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-1.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A candle is lit on a wooden table in front of a fire place with candles and other items on top of it.",
          "time": 11
        },
        "git": {
          "caption": "Two candles are lit next to each other on a table, one of them is lit up and the other is lit up.",
          "time": 23
        },
        "blip": {
          "caption": "There is a lit candle sitting on top of a wooden table next to a game board and a glass of wine on the table.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A candle sits on top of a wooden table.",
          "time": 27
        },
        "blip2-flan": {
          "caption": "A candle sits on a wooden table next to a backgammon board and a glass of wine.",
          "time": 37
        },
        "llama32-vision-11b": {
          "caption": "A dimly lit room with a wooden table, featuring a candle, a backgammon board, and other objects.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-3.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A large body of water with a bunch of birds on top of it, with a beach next to it in the distance.",
          "time": 11
        },
        "git": {
          "caption": "A large rock jutting out of the ocean is in the foreground and a couple of people are on the beach.",
          "time": 22
        },
        "blip": {
          "caption": "A group of people walking on the beach with a rock formation in the background and seagulls in the foreground.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "Haystack rock, cannon beach, oregon.",
          "time": 35
        },
        "blip2-flan": {
          "caption": "Cannon beach, oregon at sunset with people on the beach and rock formations in the foreground.",
          "time": 37
        },
        "llama32-vision-11b": {
          "caption": "A large rock formation on a beach, with people standing nearby and seagulls flying overhead.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-2.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A hotel room with a view of the ocean and skyscrapers on the other side of the clock tower in the distance.",
          "time": 12
        },
        "git": {
          "caption": "A view of a city from the window of a hotel with a sunset in the background and a cityscape in the distance.",
          "time": 23
        },
        "blip": {
          "caption": "A bedroom with a view of the city from it ' s window overlooking the water and a cityscape in the distance.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A view of the city skyline through a window in a hotel room.",
          "time": 25
        },
        "blip2-flan": {
          "caption": "A bed in a room with a view of a city skyline at dusk on a cloudy day.",
          "time": 41
        },
        "llama32-vision-11b": {
          "caption": "A bedroom with a large window overlooking a city skyline at sunset or sunrise.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-6.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A man standing on top of a skateboard with his foot on the ground next to a skateboard in the grass.",
          "time": 11
        },
        "git": {
          "caption": "A person on a skateboard that has the word tron on the side of it and the word tron on the side.",
          "time": 24
        },
        "blip": {
          "caption": "There is a man that is standing on a skateboard in the grass with his feet on the board and one foot on the back of the board.",
          "time": 9
        },
        "blip2-opt": {
          "caption": "A man riding an electric scooter on a grassy field.",
          "time": 24
        },
        "blip2-flan": {
          "caption": "A man is standing on a skateboard in a grassy area with a camper in the background.",
          "time": 40
        },
        "llama32-vision-11b": {
          "caption": "A person riding an onewheel, a type of electric skateboard, with their hands on the wheel and wearing casual clothing.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-7.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A woman sitting on a ledge in front of a building with a view of the city from the balcony, looking out into the distance.",
          "time": 11
        },
        "git": {
          "caption": "A woman with red hair is standing in front of a large window that has an american flag on the wall behind her.",
          "time": 22
        },
        "blip": {
          "caption": "A woman standing in front of a window looking out on a cityscape with skyscrapers in the background.",
          "time": 7
        },
        "blip2-opt": {
          "caption": "A red - haired woman standing in front of a large window with skyscrapers in the background.",
          "time": 30
        },
        "blip2-flan": {
          "caption": "A woman is standing in front of a window looking at a city with tall buildings and skyscrapers.",
          "time": 41
        },
        "llama32-vision-11b": {
          "caption": "A woman with red hair standing in front of a large window, looking out at a cityscape.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-5.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A large group of people sitting at a table in front of a large building with a clock on the top of it.",
          "time": 11
        },
        "git": {
          "caption": "A group of people are sitting at a table in front of a building with the word sagrada familia on it.",
          "time": 23
        },
        "blip": {
          "caption": "A group of people sitting at a table with a view of the city and mountains in the background.",
          "time": 7
        },
        "blip2-opt": {
          "caption": "A group of people sitting at an outdoor table in front of the sagrada familia in barcelona, spain.",
          "time": 41
        },
        "blip2-flan": {
          "caption": "Barcelona's sagrada familia is one of the most iconic buildings in barcelona.",
          "time": 40
        },
        "llama32-vision-11b": {
          "caption": "A cityscape with a large building and people sitting at tables, possibly enjoying drinks or food.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-4.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A living room with a mirror, candles, and a vase of flowers on a table in front of a mirror.",
          "time": 11
        },
        "git": {
          "caption": "A picture frame is hanging on a wall next to a vase and a vase with the word tulips on it.",
          "time": 28
        },
        "blip": {
          "caption": "A room with a painting on the wall and two vases on the table in front of it.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A room with a painting on the wall, a picture frame, and a chandelier.",
          "time": 30
        },
        "blip2-flan": {
          "caption": "A room with a painting on the wall and a couple of framed pictures hanging on the wall next to it.",
          "time": 39
        },
        "llama32-vision-11b": {
          "caption": "An empty gold frame on a wall with ornate wallpaper, surrounded by other decorative items.",
          "time": 43
        }
      }
    }
  ]
}

[jeremyandrews]

The error handling could be improved. I had the old-style model definition which I pasted in and got the resulting error which was rather confusing at first glance:

{
  "image": "./test-images/image-4.jpg",
  "captions": {
    "llama-90b": {
      "caption": "ERROR: 'platform'"
    }
  }
}

I then commented out the exception handler to see the full error: with that I was able to understand that the model definition format had changed. Once fixed, I successfully ran with two more local models:

 {
  "results": [
    {
      "image": "./test-images/image-9.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A wooden shrine with religious figure, surrounded by mountains and signposts.",
          "time": 6
        },
        "llama32-vision-90b": {
          "caption": "A wooden shrine with a statue of mary, situated in front of a mountainous landscape.",
          "time": 56
        }
      }
    },
    {
      "image": "./test-images/image-8.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "\"a person sitting on edge of pool with water splashing around them.",
          "time": 18
        },
        "llama32-vision-90b": {
          "caption": "A boy climbing out of the water onto a dock or pier.",
          "time": 39
        }
      }
    },
    {
      "image": "./test-images/image-10.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "\"this image shows a bustling city street at night, filled with people and illuminated by neon lights.",
          "time": 18
        },
        "llama32-vision-90b": {
          "caption": "A busy city street with many people and buildings.",
          "time": 57
        }
      }
    },
    {
      "image": "./test-images/image-11.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A laptop on a balcony, powered by solar panels, with a city skyline in the background.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A solar panel on a rooftop with a cityscape in the background.",
          "time": 45
        }
      }
    },
    {
      "image": "./test-images/image-12.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "Two shirtless men on boat, watching another person water skiing.",
          "time": 18
        },
        "llama32-vision-90b": {
          "caption": "Two men on a boat watching another man wakeboarding.",
          "time": 38
        }
      }
    },
    {
      "image": "./test-images/image-1.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A candlelit wooden table with a musical instrument and a book.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A candle and backgammon board on a wooden table.",
          "time": 39
        }
      }
    },
    {
      "image": "./test-images/image-3.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "\"this image shows a serene beach scene at sunset with silhouetted rocks and people enjoying the view.",
          "time": 18
        },
        "llama32-vision-90b": {
          "caption": "A beach scene with a large rock formation in the water and people on the shore.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-2.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A bedroom with a city skyline view at sunset.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A city skyline at sunset, viewed from inside a room with a large window.",
          "time": 40
        }
      }
    },
    {
      "image": "./test-images/image-6.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A man fixing his electric skateboard on grass.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A man preparing to ride an electric skateboard, known as a one-wheel board.",
          "time": 42
        }
      }
    },
    {
      "image": "./test-images/image-7.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A woman walking through an atrium with large windows and skyscrapers outside.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A woman with red hair standing in front of a large window, looking out at the cityscape.",
          "time": 40
        }
      }
    },
    {
      "image": "./test-images/image-5.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "\"this image shows people enjoying an outdoor dining experience with a stunning view of barcelona's skyline and sagrada familia at sunset.",
          "time": 18
        },
        "llama32-vision-90b": {
          "caption": "A cityscape with a large cathedral and mountains in the background, featuring people dining at tables on a rooftop or balcony.",
          "time": 42
        }
      }
    },
    {
      "image": "./test-images/image-4.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "\"this image shows an ornate room with framed pictures on the wall and a chandelier hanging from the ceiling.",
          "time": 18
        },
        "llama32-vision-90b": {
          "caption": "An empty gold frame on a wall with two paintings and a candelabra.",
          "time": 42
        }
      }
    }
  ]
}

[dbuytaert]

Thank you for running those again. As expected, your larger models continue to produce better captions compared to my smaller models.

However, I noticed that some captions start off wrong, which suggests that clean_caption isn't working as intended. Here are a couple of examples of where the caption starts of incorrectly:

• "\"a person sitting on edge of pool with water splashing around them."
• "\"this image shows a bustling city street at night, filled with people and illuminated by neon lights."

The clean_caption function uses regular expressions to improve captions by removing unnecessary prefixes like "this image shows". According to accessibility best practices, such phrases are redundant since screen readers already inform the user that they are interacting with an image. Removing those phrases saves some valuable time. Clearly, that didn't work as expected, and an extra quotation mark was added as well.

To help debug this issue, could you make sure you're using the latest version of the code? Then, try commenting out the regular expressions in clean_caption and run everything again? This will allow us to see the original captions. From there, I could create a unit test that takes the original captions as input to figure out where clean_caption goes wrong. Of course, if regex are your thing, feel free to roll up your sleeves and dive in!

Also, if you copy-paste your model descriptions, I can add them to the script. Makes it easier for you going forward.

[jeremyandrews]

I ran this before I saw your comment here -- I just uncommented all your debug output:

jandrews@Jeremys-MacBook-Pro-2 image-caption % find . -name "*.jpg" -exec ./caption {} --time \;       
Original: "a wooden shrine with religious figures and signs, set against a mountainous backdrop
After pattern #1: "a wooden shrine with religious figures and signs, set against a mountainous backdrop
After pattern #2: "a wooden shrine with religious figures and signs, set against a mountainous backdrop
Original: this image shows a wooden shrine with a statue of mary and a signpost pointing to various locations, including rundweg st
After pattern #1: this image shows a wooden shrine with a statue of mary and a signpost pointing to various locations, including rundweg st
After pattern #2: a wooden shrine with a statue of mary and a signpost pointing to various locations, including rundweg st
{
  "image": "./test-images/image-9.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "\"a wooden shrine with religious figures and signs, set against a mountainous backdrop.",
      "time": 6
    },
    "llama32-vision-90b": {
      "caption": "A wooden shrine with a statue of mary and a signpost pointing to various locations, including rundweg st.",
      "time": 57
    }
  }
}
Original: "a person sitting on edge of pool with water splashing around them, wearing colorful swim trunks and no shirt
After pattern #1: "a person sitting on edge of pool with water splashing around them, wearing colorful swim trunks and no shirt
After pattern #2: "a person sitting on edge of pool with water splashing around them, wearing colorful swim trunks and no shirt
Original: this image shows a person climbing out of the water onto a dock or boat, with their legs and lower body visible as they ascend from the water's surface
After pattern #1: this image shows a person climbing out of the water onto a dock or boat, with their legs and lower body visible as they ascend from the water's surface
After pattern #2: a person climbing out of the water onto a dock or boat, with their legs and lower body visible as they ascend from the water's surface
{
  "image": "./test-images/image-8.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "\"a person sitting on edge of pool with water splashing around them, wearing colorful swim trunks and no shirt.",
      "time": 17
    },
    "llama32-vision-90b": {
      "caption": "A person climbing out of the water onto a dock or boat, with their legs and lower body visible as they ascend from the water's surface.",
      "time": 56
    }
  }
}
Original: "this image shows a bustling city street at night, filled with people and illuminated by neon lights
After pattern #1: "this image shows a bustling city street at night, filled with people and illuminated by neon lights
After pattern #2: "this image shows a bustling city street at night, filled with people and illuminated by neon lights
Original: this image shows a busy city street with many people and buildings
After pattern #1: this image shows a busy city street with many people and buildings
After pattern #2: a busy city street with many people and buildings
{
  "image": "./test-images/image-10.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "\"this image shows a bustling city street at night, filled with people and illuminated by neon lights.",
      "time": 17
    },
    "llama32-vision-90b": {
      "caption": "A busy city street with many people and buildings.",
      "time": 44
    }
  }
}
Original: a laptop on a balcony, powered by solar panels, with a city skyline in the background
After pattern #1: a laptop on a balcony, powered by solar panels, with a city skyline in the background
After pattern #2: a laptop on a balcony, powered by solar panels, with a city skyline in the background
Original: this image shows a solar panel on a rooftop with a city skyline in the background
After pattern #1: this image shows a solar panel on a rooftop with a city skyline in the background
After pattern #2: a solar panel on a rooftop with a city skyline in the background
{
  "image": "./test-images/image-11.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "A laptop on a balcony, powered by solar panels, with a city skyline in the background.",
      "time": 17
    },
    "llama32-vision-90b": {
      "caption": "A solar panel on a rooftop with a city skyline in the background.",
      "time": 39
    }
  }
}
Original: two shirtless men on boat, watching another person water skiing
After pattern #1: two shirtless men on boat, watching another person water skiing
After pattern #2: two shirtless men on boat, watching another person water skiing
Original: this image shows two men on a boat watching another man wakeboarding
After pattern #1: this image shows two men on a boat watching another man wakeboarding
After pattern #2: two men on a boat watching another man wakeboarding
{
  "image": "./test-images/image-12.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "Two shirtless men on boat, watching another person water skiing.",
      "time": 16
    },
    "llama32-vision-90b": {
      "caption": "Two men on a boat watching another man wakeboarding.",
      "time": 40
    }
  }
}
Original: "a candlelit wooden table with musical instruments and a cozy ambiance
After pattern #1: "a candlelit wooden table with musical instruments and a cozy ambiance
After pattern #2: "a candlelit wooden table with musical instruments and a cozy ambiance
Original: this image shows a candle on a wooden table with a backgammon board and two glasses of wine
After pattern #1: this image shows a candle on a wooden table with a backgammon board and two glasses of wine
After pattern #2: a candle on a wooden table with a backgammon board and two glasses of wine
{
  "image": "./test-images/image-1.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "\"a candlelit wooden table with musical instruments and a cozy ambiance.",
      "time": 16
    },
    "llama32-vision-90b": {
      "caption": "A candle on a wooden table with a backgammon board and two glasses of wine.",
      "time": 41
    }
  }
}
Original: "this image shows a serene beach scene at sunset with silhouetted rocks and people enjoying the view
After pattern #1: "this image shows a serene beach scene at sunset with silhouetted rocks and people enjoying the view
After pattern #2: "this image shows a serene beach scene at sunset with silhouetted rocks and people enjoying the view
Original: this image shows a beach scene with people on the sand and large rocks in the water
After pattern #1: this image shows a beach scene with people on the sand and large rocks in the water
After pattern #2: a beach scene with people on the sand and large rocks in the water
{
  "image": "./test-images/image-3.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "\"this image shows a serene beach scene at sunset with silhouetted rocks and people enjoying the view.",
      "time": 17
    },
    "llama32-vision-90b": {
      "caption": "A beach scene with people on the sand and large rocks in the water.",
      "time": 38
    }
  }
}
Original: this image shows an interior view of a hotel room at sunset, with a city skyline visible through large windows
After pattern #1: this image shows an interior view of a hotel room at sunset, with a city skyline visible through large windows
After pattern #2: an interior view of a hotel room at sunset, with a city skyline visible through large windows
Original: this image shows a city skyline at sunset, viewed from inside a room with a large window
After pattern #1: this image shows a city skyline at sunset, viewed from inside a room with a large window
After pattern #2: a city skyline at sunset, viewed from inside a room with a large window
{
  "image": "./test-images/image-2.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "An interior view of a hotel room at sunset, with a city skyline visible through large windows.",
      "time": 17
    },
    "llama32-vision-90b": {
      "caption": "A city skyline at sunset, viewed from inside a room with a large window.",
      "time": 40
    }
  }
}
Original: a man fixing an electric skateboard on grass
After pattern #1: a man fixing an electric skateboard on grass
After pattern #2: a man fixing an electric skateboard on grass
Original: this image shows a man preparing to ride an electric skateboard, known as a one-wheel board
After pattern #1: this image shows a man preparing to ride an electric skateboard, known as a one-wheel board
After pattern #2: a man preparing to ride an electric skateboard, known as a one-wheel board
{
  "image": "./test-images/image-6.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "A man fixing an electric skateboard on grass.",
      "time": 16
    },
    "llama32-vision-90b": {
      "caption": "A man preparing to ride an electric skateboard, known as a one-wheel board.",
      "time": 40
    }
  }
}
Original: a woman walking through an open, modern office space with large windows and city views
After pattern #1: a woman walking through an open, modern office space with large windows and city views
After pattern #2: a woman walking through an open, modern office space with large windows and city views
Original: this image shows a woman with red hair standing in front of a large window, looking out at the cityscape
After pattern #1: this image shows a woman with red hair standing in front of a large window, looking out at the cityscape
After pattern #2: a woman with red hair standing in front of a large window, looking out at the cityscape
{
  "image": "./test-images/image-7.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "A woman walking through an open, modern office space with large windows and city views.",
      "time": 17
    },
    "llama32-vision-90b": {
      "caption": "A woman with red hair standing in front of a large window, looking out at the cityscape.",
      "time": 41
    }
  }
}
Original: "this image shows people enjoying an outdoor dining experience with a stunning cityscape and sunset view, including the iconic sagrada familia cathedral
After pattern #1: "this image shows people enjoying an outdoor dining experience with a stunning cityscape and sunset view, including the iconic sagrada familia cathedral
After pattern #2: "this image shows people enjoying an outdoor dining experience with a stunning cityscape and sunset view, including the iconic sagrada familia cathedral
Original: this image shows a cityscape with a large cathedral and mountains in the background, featuring people dining at tables on a rooftop or balcony
After pattern #1: this image shows a cityscape with a large cathedral and mountains in the background, featuring people dining at tables on a rooftop or balcony
After pattern #2: a cityscape with a large cathedral and mountains in the background, featuring people dining at tables on a rooftop or balcony
{
  "image": "./test-images/image-5.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "\"this image shows people enjoying an outdoor dining experience with a stunning cityscape and sunset view, including the iconic sagrada familia cathedral.",
      "time": 18
    },
    "llama32-vision-90b": {
      "caption": "A cityscape with a large cathedral and mountains in the background, featuring people dining at tables on a rooftop or balcony.",
      "time": 43
    }
  }
}
Original: "this image shows an ornate room with framed pictures on the wall and a lit candle sconce
After pattern #1: "this image shows an ornate room with framed pictures on the wall and a lit candle sconce
After pattern #2: "this image shows an ornate room with framed pictures on the wall and a lit candle sconce
Original: this image shows an empty gold frame on a wall with a painting to its left and right, accompanied by a lit candelabra
After pattern #1: this image shows an empty gold frame on a wall with a painting to its left and right, accompanied by a lit candelabra
After pattern #2: an empty gold frame on a wall with a painting to its left and right, accompanied by a lit candelabra
{
  "image": "./test-images/image-4.jpg",
  "captions": {
    "llava:34b-v16-q8": {
      "caption": "\"this image shows an ornate room with framed pictures on the wall and a lit candle sconce.",
      "time": 17
    },
    "llama32-vision-90b": {
      "caption": "An empty gold frame on a wall with a painting to its left and right, accompanied by a lit candelabra.",
      "time": 58
    }
  }
}

[jeremyandrews]

These are the models I'm currently using:

   "llava:34b-v16-q8": {
       "platform": "ollama",
       "architecture": "llava:34b",
       "path": "llava:34b-v1.6-q8_0",
       "description": "Llaa 34b (90B, Q8)",
       "date": "2024"
   },
   "llama32-vision-90b": {
       "platform": "ollama",
       "architecture": "llama3.2-vision",
       "path": "llama3.2-vision:90b-instruct-q8_0",
       "description": "Llama 3.2 Vision (90B, Q8)",
       "date": "2024"
   }

[dbuytaert]

Thank you! I noticed that different models output captions in different formats: one of your models add quotes around the caption, while mine don't. This shows up in your test output.

The clean_caption function assumed there were no quotes, causing issues like "a wooden shrine ..." instead of A wooden shrine ....

I updated clean_caption to strip the quotes and added a first unit test to verify the fix. Could you give it another try?

Another option is to modify your model to stop adding quotes, but that might be more difficult.

[jeremyandrews]

Another option is to modify your model to stop adding quotes, but that might be more difficult.

I'm curious how you were thinking we'd do that? You mean an extra regex command specific to the model, or something else?

[jeremyandrews]

I've run it again with the updated code:

{
  "results": [
    {
      "image": "./test-images/image-9.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A wooden shrine with religious figures on a mountain trail.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A statue of mary in a wooden shrine with a signpost pointing to various locations, including rundweg st.",
          "time": 60
        }
      }
    },
    {
      "image": "./test-images/image-8.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A person sitting on edge of pool with water splashing around them.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A person climbing out of the water onto a dock or boat, with their legs and lower body visible as they ascend from the water's surface.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-10.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A bustling city street at night, filled with people and illuminated by neon lights.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A bustling city street with many people and buildings.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-11.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A laptop on a balcony, powered by solar panels, overlooking a city skyline at sunset.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A solar panel on a rooftop with a cityscape in the background.",
          "time": 38
        }
      }
    },
    {
      "image": "./test-images/image-12.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "Two shirtless men on boat, watching another person water skiing.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "Two men on a boat watching another man wakeboarding.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-1.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A dimly lit room with a wooden table holding a candle and a musical instrument.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A candle and backgammon board on a wooden table.",
          "time": 40
        }
      }
    },
    {
      "image": "./test-images/image-3.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A serene beach scene at sunset with people enjoying the view and birds flying over the water.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A beach scene with people on the sand and large rocks in the water.",
          "time": 40
        }
      }
    },
    {
      "image": "./test-images/image-2.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A bedroom with a stunning city skyline view at sunset.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A city skyline at sunset, viewed from inside a room with a large window.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-6.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A man fixing his electric skateboard on grass.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A man preparing to ride an electric skateboard, known as a one-wheel board.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-7.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "A woman walking past large windows with cityscape view.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A woman with red hair standing in front of a large window, looking out at the cityscape.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-5.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "People enjoying an outdoor dining experience with a stunning view of barcelona's skyline and sagrada familia at sunset.",
          "time": 18
        },
        "llama32-vision-90b": {
          "caption": "A cityscape with people dining at tables on an outdoor patio, overlooking the city skyline.",
          "time": 42
        }
      }
    },
    {
      "image": "./test-images/image-4.jpg",
      "captions": {
        "llava:34b-v16-q8": {
          "caption": "An ornate room with framed pictures on the wall and a lit candle sconce.",
          "time": 18
        },
        "llama32-vision-90b": {
          "caption": "An ornate room with a gold frame, a painting of a man, and a vase on a table.",
          "time": 41
        }
      }
    }
  ]
}

[dbuytaert]

The updated clean_caption seems to be working as expected as the captions in your latest JSON file are formatted correctly.

I've combined both of our JSON files and merged all the captions:

{
  "results": [
    {
      "image": "./test-images/image-1.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A candle is lit on a wooden table in front of a fire place with candles and other items on top of it.",
          "time": 11
        },
        "git": {
          "caption": "Two candles are lit next to each other on a table, one of them is lit up and the other is lit up.",
          "time": 23
        },
        "blip": {
          "caption": "There is a lit candle sitting on top of a wooden table next to a game board and a glass of wine on the table.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A candle sits on top of a wooden table.",
          "time": 27
        },
        "blip2-flan": {
          "caption": "A candle sits on a wooden table next to a backgammon board and a glass of wine.",
          "time": 37
        },
        "llama32-vision-11b": {
          "caption": "A dimly lit room with a wooden table, featuring a candle, a backgammon board, and other objects.",
          "time": 43
        },
        "llava-34b": {
          "caption": "A dimly lit room with a wooden table holding a candle and a musical instrument.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A candle and backgammon board on a wooden table.",
          "time": 40
        }
      }
    },
    {
      "image": "./test-images/image-2.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A hotel room with a view of the ocean and skyscrapers on the other side of the clock tower in the distance.",
          "time": 12
        },
        "git": {
          "caption": "A view of a city from the window of a hotel with a sunset in the background and a cityscape in the distance.",
          "time": 23
        },
        "blip": {
          "caption": "A bedroom with a view of the city from it ' s window overlooking the water and a cityscape in the distance.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A view of the city skyline through a window in a hotel room.",
          "time": 25
        },
        "blip2-flan": {
          "caption": "A bed in a room with a view of a city skyline at dusk on a cloudy day.",
          "time": 41
        },
        "llama32-vision-11b": {
          "caption": "A bedroom with a large window overlooking a city skyline at sunset or sunrise.",
          "time": 43
        },
        "llava-34b": {
          "caption": "A bedroom with a stunning city skyline view at sunset.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A city skyline at sunset, viewed from inside a room with a large window.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-3.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A large body of water with a bunch of birds on top of it, with a beach next to it in the distance.",
          "time": 11
        },
        "git": {
          "caption": "A large rock jutting out of the ocean is in the foreground and a couple of people are on the beach.",
          "time": 22
        },
        "blip": {
          "caption": "A group of people walking on the beach with a rock formation in the background and seagulls in the foreground.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "Haystack rock, cannon beach, oregon.",
          "time": 35
        },
        "blip2-flan": {
          "caption": "Cannon beach, oregon at sunset with people on the beach and rock formations in the foreground.",
          "time": 37
        },
        "llama32-vision-11b": {
          "caption": "A large rock formation on a beach, with people standing nearby and seagulls flying overhead.",
          "time": 43
        },
        "llava-34b": {
          "caption": "A serene beach scene at sunset with people enjoying the view and birds flying over the water.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A beach scene with people on the sand and large rocks in the water.",
          "time": 40
        }
      }
    },
    {
      "image": "./test-images/image-4.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A living room with a mirror, candles, and a vase of flowers on a table in front of a mirror.",
          "time": 11
        },
        "git": {
          "caption": "A picture frame is hanging on a wall next to a vase and a vase with the word tulips on it.",
          "time": 28
        },
        "blip": {
          "caption": "A room with a painting on the wall and two vases on the table in front of it.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A room with a painting on the wall, a picture frame, and a chandelier.",
          "time": 30
        },
        "blip2-flan": {
          "caption": "A room with a painting on the wall and a couple of framed pictures hanging on the wall next to it.",
          "time": 39
        },
        "llama32-vision-11b": {
          "caption": "An empty gold frame on a wall with ornate wallpaper, surrounded by other decorative items.",
          "time": 43
        },
        "llava-34b": {
          "caption": "An ornate room with framed pictures on the wall and a lit candle sconce.",
          "time": 18
        },
        "llama32-vision-90b": {
          "caption": "An ornate room with a gold frame, a painting of a man, and a vase on a table.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-5.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A large group of people sitting at a table in front of a large building with a clock on the top of it.",
          "time": 11
        },
        "git": {
          "caption": "A group of people are sitting at a table in front of a building with the word sagrada familia on it.",
          "time": 23
        },
        "blip": {
          "caption": "A group of people sitting at a table with a view of the city and mountains in the background.",
          "time": 7
        },
        "blip2-opt": {
          "caption": "A group of people sitting at an outdoor table in front of the sagrada familia in barcelona, spain.",
          "time": 41
        },
        "blip2-flan": {
          "caption": "Barcelona's sagrada familia is one of the most iconic buildings in barcelona.",
          "time": 40
        },
        "llama32-vision-11b": {
          "caption": "A cityscape with a large building and people sitting at tables, possibly enjoying drinks or food.",
          "time": 43
        },
        "llava-34b": {
          "caption": "People enjoying an outdoor dining experience with a stunning view of barcelona's skyline and sagrada familia at sunset.",
          "time": 18
        },
        "llama32-vision-90b": {
          "caption": "A cityscape with people dining at tables on an outdoor patio, overlooking the city skyline.",
          "time": 42
        }
      }
    },
    {
      "image": "./test-images/image-6.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A man standing on top of a skateboard with his foot on the ground next to a skateboard in the grass.",
          "time": 11
        },
        "git": {
          "caption": "A person on a skateboard that has the word tron on the side of it and the word tron on the side.",
          "time": 24
        },
        "blip": {
          "caption": "There is a man that is standing on a skateboard in the grass with his feet on the board and one foot on the back of the board.",
          "time": 9
        },
        "blip2-opt": {
          "caption": "A man riding an electric scooter on a grassy field.",
          "time": 24
        },
        "blip2-flan": {
          "caption": "A man is standing on a skateboard in a grassy area with a camper in the background.",
          "time": 40
        },
        "llama32-vision-11b": {
          "caption": "A person riding an onewheel, a type of electric skateboard, with their hands on the wheel and wearing casual clothing.",
          "time": 43
        },
        "llava-34b": {
          "caption": "A man fixing his electric skateboard on grass.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A man preparing to ride an electric skateboard, known as a one-wheel board.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-7.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A woman sitting on a ledge in front of a building with a view of the city from the balcony, looking out into the distance.",
          "time": 11
        },
        "git": {
          "caption": "A woman with red hair is standing in front of a large window that has an american flag on the wall behind her.",
          "time": 22
        },
        "blip": {
          "caption": "A woman standing in front of a window looking out on a cityscape with skyscrapers in the background.",
          "time": 7
        },
        "blip2-opt": {
          "caption": "A red - haired woman standing in front of a large window with skyscrapers in the background.",
          "time": 30
        },
        "blip2-flan": {
          "caption": "A woman is standing in front of a window looking at a city with tall buildings and skyscrapers.",
          "time": 41
        },
        "llama32-vision-11b": {
          "caption": "A woman with red hair standing in front of a large window, looking out at a cityscape.",
          "time": 43
        },
        "llava-34b": {
          "caption": "A woman walking past large windows with cityscape view.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A woman with red hair standing in front of a large window, looking out at the cityscape.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-8.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A man in a bathing suit is sitting on a boogie board at the water's edge with his feet propped up.",
          "time": 11
        },
        "git": {
          "caption": "A person is standing in the water and one of them is wearing a pair of shorts with the words \" i love you \" on it.",
          "time": 26
        },
        "blip": {
          "caption": "There is a man that is standing on the edge of a swimming pool with his feet in the water and a hose attached to the side of the pool.",
          "time": 7
        },
        "blip2-opt": {
          "caption": "A man in swim trunks standing on the edge of a pool.",
          "time": 26
        },
        "blip2-flan": {
          "caption": "A young boy is leaning on the railing of a dock in a body of water with his feet in the water.",
          "time": 40
        },
        "llama32-vision-11b": {
          "caption": "A person standing on a dock, with their legs submerged in water and holding onto a metal archway.",
          "time": 46
        },
        "llava-34b": {
          "caption": "A person sitting on edge of pool with water splashing around them.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A person climbing out of the water onto a dock or boat, with their legs and lower body visible as they ascend from the water's surface.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-9.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A bird perched on top of a wooden post in front of a mountain range with a view of a mountain range and mountains.",
          "time": 10
        },
        "git": {
          "caption": "A wooden sign in front of a mountain with a statue on it that says \" meise museum \" on the top.",
          "time": 21
        },
        "blip": {
          "caption": "There is a birdhouse with a statue on top of it in the middle of a field with trees and mountains in the background.",
          "time": 7
        },
        "blip2-opt": {
          "caption": "A statue of mary on a wooden post in the middle of a field with mountains in the background.",
          "time": 34
        },
        "blip2-flan": {
          "caption": "A statue of the virgin mary sits on top of a wooden post in the middle of a mountain.",
          "time": 39
        },
        "llama32-vision-11b": {
          "caption": "A wooden shrine with a statue of mary and a signpost in front of mountains.",
          "time": 43
        },
        "llava-34b": {
          "caption": "A wooden shrine with religious figures on a mountain trail.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A statue of mary in a wooden shrine with a signpost pointing to various locations, including rundweg st.",
          "time": 60
        }
      }
    },
    {
      "image": "./test-images/image-10.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A city at night with skyscrapers and a traffic light on the side of the street in front of a tall building.",
          "time": 12
        },
        "git": {
          "caption": "A busy city street is lit up at night, with the word qroi on the right side of the sign.",
          "time": 22
        },
        "blip": {
          "caption": "This is an aerial view of a busy city street at night with lots of people walking and cars on the side of the road.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "An aerial view of a busy city street at night.",
          "time": 32
        },
        "blip2-flan": {
          "caption": "An aerial view of a busy street in tokyo, japanese city at night with large billboards.",
          "time": 41
        },
        "llama32-vision-11b": {
          "caption": "A bustling city street at night, with numerous billboards and advertisements lining the buildings.",
          "time": 43
        },
        "llava-34b": {
          "caption": "A bustling city street at night, filled with people and illuminated by neon lights.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A bustling city street with many people and buildings.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-11.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A building that has a lot of windows on top of it and some cars parked on the side of the road in front of it.",
          "time": 10
        },
        "git": {
          "caption": "A laptop sits on a table in front of a cityscape, with the skyline in the background and a cityscape in the background.",
          "time": 24
        },
        "blip": {
          "caption": "There is a laptop sitting on top of a table on a balcony with a cityscape in the background and a solar panel on top of it.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A solar panel on top of a table in front of a city skyline.",
          "time": 27
        },
        "blip2-flan": {
          "caption": "A solar panel sits on top of a railing on a balcony overlooking a city skyline.",
          "time": 40
        },
        "llama32-vision-11b": {
          "caption": "A solar panel on top of a table with a city skyline in the background.",
          "time": 45
        },
        "llava-34b": {
          "caption": "A laptop on a balcony, powered by solar panels, overlooking a city skyline at sunset.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A solar panel on a rooftop with a cityscape in the background.",
          "time": 38
        }
      }
    },
    {
      "image": "./test-images/image-12.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A man standing on top of a boat next to another man holding a surfboard in one hand and a surfboard in the other.",
          "time": 11
        },
        "git": {
          "caption": "Two men are in a boat, one of them is wearing an orange hat and the other is wearing an orange hat.",
          "time": 21
        },
        "blip": {
          "caption": "There are two men riding on the back of a boat in the water, one of them is on a surfboard and the other is on a board.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "Three young men sitting on the back of a boat.",
          "time": 26
        },
        "blip2-flan": {
          "caption": "A group of people sitting on a boat watching a man ride a surfboard in the middle of the water.",
          "time": 38
        },
        "llama32-vision-11b": {
          "caption": "Two shirtless men on a boat watching people wakeboarding or surfing behind it.",
          "time": 43
        },
        "llava-34b": {
          "caption": "Two shirtless men on boat, watching another person water skiing.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "Two men on a boat watching another man wakeboarding.",
          "time": 41
        }
      }
    }
  ]
}

[dbuytaert]

I was able to run llava-34b on my own laptop, as well as llava-13b. I added these, and updated the merged file:

{
  "results": [
    {
      "image": "./test-images/image-1.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A candle is lit on a wooden table in front of a fire place with candles and other items on top of it.",
          "time": 11
        },
        "git": {
          "caption": "Two candles are lit next to each other on a table, one of them is lit up and the other is lit up.",
          "time": 24
        },
        "blip": {
          "caption": "There is a lit candle sitting on top of a wooden table next to a game board and a glass of wine on the table.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A candle sits on top of a wooden table.",
          "time": 35
        },
        "blip2-flan": {
          "caption": "A candle sits on a wooden table next to a backgammon board and a glass of wine.",
          "time": 43
        },
        "llama32-vision-11b": {
          "caption": "A dimly lit room with a wooden table, featuring a backgammon board and two candles.",
          "time": 44
        },
        "llava-13b": {
          "caption": "A cozy indoor setting with candles, a guitar, and a warm ambiance.",
          "time": 8
        },
        "llava-34b": {
          "caption": "A candlelit wooden table with musical instruments and a cozy ambiance.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A candle and backgammon board on a wooden table.",
          "time": 40
        }
      }
    },
    {
      "image": "./test-images/image-2.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A hotel room with a view of the ocean and skyscrapers on the other side of the clock tower in the distance.",
          "time": 11
        },
        "git": {
          "caption": "A view of a city from the window of a hotel with a sunset in the background and a cityscape in the distance.",
          "time": 23
        },
        "blip": {
          "caption": "A bedroom with a view of the city from it ' s window overlooking the water and a cityscape in the distance.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A view of the city skyline through a window in a hotel room.",
          "time": 35
        },
        "blip2-flan": {
          "caption": "A bed in a room with a view of a city skyline at dusk on a cloudy day.",
          "time": 44
        },
        "llama32-vision-11b": {
          "caption": "A city skyline viewed through a window from a darkened bedroom, with a bed and lamp visible in the foreground.",
          "time": 44
        },
        "llava-13b": {
          "caption": "A city skyline at sunset, viewed from a hotel room with a large window.",
          "time": 8
        },
        "llava-34b": {
          "caption": "A room with a large window overlooking a city skyline at sunset.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A city skyline at sunset, viewed from inside a room with a large window.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-3.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A large body of water with a bunch of birds on top of it, with a beach next to it in the distance.",
          "time": 11
        },
        "git": {
          "caption": "A large rock jutting out of the ocean is in the foreground and a couple of people are on the beach.",
          "time": 23
        },
        "blip": {
          "caption": "A group of people walking on a beach near the ocean with a large rock formation in the background.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "Haystack rock, cannon beach, oregon.",
          "time": 47
        },
        "blip2-flan": {
          "caption": "Cannon beach, oregon at sunset with people on the beach and rock formations in the foreground.",
          "time": 38
        },
        "llama32-vision-11b": {
          "caption": "A large rock formation on a beach, with people standing nearby and seagulls flying overhead.",
          "time": 44
        },
        "llava-13b": {
          "caption": "A serene beach scene at sunset with a large rock formation in the background.",
          "time": 8
        },
        "llava-34b": {
          "caption": "A serene beach scene with a rocky coastline and distant islands at sunset.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A beach scene with people on the sand and large rocks in the water.",
          "time": 40
        }
      }
    },
    {
      "image": "./test-images/image-4.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A living room with a mirror, candles, and a vase of flowers on a table in front of a mirror.",
          "time": 12
        },
        "git": {
          "caption": "A picture frame is hanging on a wall next to a vase and a vase with the word tulips on it.",
          "time": 29
        },
        "blip": {
          "caption": "A room with a painting on the wall and two vases on the table in front of it.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A room with a painting on the wall, a picture frame, and a chandelier.",
          "time": 36
        },
        "blip2-flan": {
          "caption": "A room with a painting on the wall and a couple of framed pictures hanging on the wall next to it.",
          "time": 44
        },
        "llama32-vision-11b": {
          "caption": "An empty gold frame on a wall with ornate wallpaper and other decorative items.",
          "time": 43
        },
        "llava-13b": {
          "caption": "An empty wall with ornate decorations, including candlesticks and framed artwork, suggesting a formal or historical setting.",
          "time": 8
        },
        "llava-34b": {
          "caption": "An ornate room with framed pictures on the wall, a chandelier hanging from the ceiling, and a fireplace mantel adorned with decorative items.",
          "time": 19
        },
        "llama32-vision-90b": {
          "caption": "An ornate room with a gold frame, a painting of a man, and a vase on a table.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-5.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A large group of people sitting at a table in front of a large building with a clock on the top of it.",
          "time": 11
        },
        "git": {
          "caption": "A group of people are sitting at a table in front of a building with the word sagrada familia on it.",
          "time": 25
        },
        "blip": {
          "caption": "A group of people sitting at a table with a view of the city and mountains in the background.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A group of people sitting at an outdoor table in front of the sagrada familia in barcelona, spain.",
          "time": 45
        },
        "blip2-flan": {
          "caption": "Barcelona's sagrada familia is one of the most iconic buildings in barcelona.",
          "time": 47
        },
        "llama32-vision-11b": {
          "caption": "A cityscape with a large building and people sitting at tables, possibly during sunset or sunrise.",
          "time": 44
        },
        "llava-13b": {
          "caption": "A cityscape at sunset with people enjoying an outdoor dining experience, featuring the iconic sagrada familia church in the distance.",
          "time": 8
        },
        "llava-34b": {
          "caption": "People enjoying an outdoor dining experience with a stunning cityscape and sunset view.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A cityscape with people dining at tables on an outdoor patio, overlooking the city skyline.",
          "time": 42
        }
      }
    },
    {
      "image": "./test-images/image-6.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A man standing on top of a skateboard with his foot on the ground next to a skateboard in the grass.",
          "time": 11
        },
        "git": {
          "caption": "A person on a skateboard that has the word tron on the side of it and the word tron on the side.",
          "time": 24
        },
        "blip": {
          "caption": "There is a man that is standing on a skateboard in the grass with his feet on the board and one foot on the back of the board.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A man riding an electric scooter on a grassy field.",
          "time": 35
        },
        "blip2-flan": {
          "caption": "A man is standing on a skateboard in a grassy area with a camper in the background.",
          "time": 46
        },
        "llama32-vision-11b": {
          "caption": "A person riding an onewheel, a type of electric skateboard, with their hands on the wheel and wearing casual clothing.",
          "time": 44
        },
        "llava-13b": {
          "caption": "A man adjusting his skateboard with a motor attached to it, outdoors on grassy terrain.",
          "time": 8
        },
        "llava-34b": {
          "caption": "A man adjusting his electric skateboard.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A man preparing to ride an electric skateboard, known as a one-wheel board.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-7.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A woman sitting on a ledge in front of a building with a view of the city from the balcony, looking out into the distance.",
          "time": 11
        },
        "git": {
          "caption": "A woman with red hair is standing in front of a large window that has an american flag on the wall behind her.",
          "time": 22
        },
        "blip": {
          "caption": "A woman standing in front of a window looking out on a cityscape with skyscrapers in the background.",
          "time": 7
        },
        "blip2-opt": {
          "caption": "A red - haired woman standing in front of a large window with skyscrapers in the background.",
          "time": 35
        },
        "blip2-flan": {
          "caption": "A woman is standing in front of a window looking at a city with tall buildings and skyscrapers.",
          "time": 40
        },
        "llama32-vision-11b": {
          "caption": "A woman with red hair standing in front of a large window, looking out at a cityscape.",
          "time": 44
        },
        "llava-13b": {
          "caption": "A woman looking out of a high-rise office window, contemplating her thoughts with a cityscape view behind her.",
          "time": 8
        },
        "llava-34b": {
          "caption": "A woman walking past a large window with a city view.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A woman with red hair standing in front of a large window, looking out at the cityscape.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-8.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A man in a bathing suit is sitting on a boogie board at the water's edge with his feet propped up.",
          "time": 12
        },
        "git": {
          "caption": "A person is standing in the water and one of them is wearing a pair of shorts with the words \" i love you \" on it.",
          "time": 28
        },
        "blip": {
          "caption": "There is a man that is standing on the edge of a swimming pool with his feet in the water and a hose attached to the side of the pool.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A man in swim trunks standing on the edge of a pool.",
          "time": 38
        },
        "blip2-flan": {
          "caption": "A young boy is leaning on the railing of a dock in a body of water with his feet in the water.",
          "time": 44
        },
        "llama32-vision-11b": {
          "caption": "A person standing on a dock, with their legs submerged in water and holding onto a metal archway.",
          "time": 44
        },
        "llava-13b": {
          "caption": "A young boy playing in water, wearing colorful swim trunks and leaning on a metal railing.",
          "time": 9
        },
        "llava-34b": {
          "caption": "A person sitting on edge of pool with water splashing around them.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A person climbing out of the water onto a dock or boat, with their legs and lower body visible as they ascend from the water's surface.",
          "time": 43
        }
      }
    },
    {
      "image": "./test-images/image-9.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A bird perched on top of a wooden post in front of a mountain range with a view of a mountain range and mountains.",
          "time": 12
        },
        "git": {
          "caption": "A wooden sign in front of a mountain with a statue on it that says \" meise museum \" on the top.",
          "time": 37
        },
        "blip": {
          "caption": "There is a birdhouse with a statue on top of it in the middle of a field with trees and mountains in the background.",
          "time": 9
        },
        "blip2-opt": {
          "caption": "A statue of mary on a wooden post in the middle of a field with mountains in the background.",
          "time": 45
        },
        "blip2-flan": {
          "caption": "A statue of the virgin mary sits on top of a wooden post in the middle of a mountain.",
          "time": 47
        },
        "llama32-vision-11b": {
          "caption": "A wooden shrine with a statue of mary and a signpost pointing to various locations, set against a mountainous backdrop.",
          "time": 45
        },
        "llava-13b": {
          "caption": "A picturesque scene of a small wooden chapel with a statue of jesus, set against a backdrop of majestic mountains and lush greenery.",
          "time": 8
        },
        "llava-34b": {
          "caption": "A wooden shrine with religious iconography, surrounded by alpine scenery and signposts.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A statue of mary in a wooden shrine with a signpost pointing to various locations, including rundweg st.",
          "time": 60
        }
      }
    },
    {
      "image": "./test-images/image-10.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A city at night with skyscrapers and a traffic light on the side of the street in front of a tall building.",
          "time": 11
        },
        "git": {
          "caption": "A busy city street is lit up at night, with the word qroi on the right side of the sign.",
          "time": 23
        },
        "blip": {
          "caption": "This is an aerial view of a busy city street at night with lots of people walking and cars on the side of the road.",
          "time": 7
        },
        "blip2-opt": {
          "caption": "An aerial view of a busy city street at night.",
          "time": 37
        },
        "blip2-flan": {
          "caption": "An aerial view of a busy street in tokyo, japanese city at night with large billboards.",
          "time": 40
        },
        "llama32-vision-11b": {
          "caption": "A bustling city street at night, with numerous billboards and advertisements lining the buildings.",
          "time": 44
        },
        "llava-13b": {
          "caption": "A bustling nighttime scene of tokyo's shibuya crossing, with neon lights and people walking across the intersection.",
          "time": 8
        },
        "llava-34b": {
          "caption": "A bustling city street at night, with illuminated buildings and numerous pedestrians.",
          "time": 17
        },
        "llama32-vision-90b": {
          "caption": "A bustling city street with many people and buildings.",
          "time": 41
        }
      }
    },
    {
      "image": "./test-images/image-11.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A building that has a lot of windows on top of it and some cars parked on the side of the road in front of it.",
          "time": 11
        },
        "git": {
          "caption": "A laptop sits on a table in front of a cityscape, with the skyline in the background and a cityscape in the background.",
          "time": 25
        },
        "blip": {
          "caption": "There is a laptop sitting on top of a table on a balcony with a cityscape in the background and a solar panel on top of it.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "A solar panel on top of a table in front of a city skyline.",
          "time": 34
        },
        "blip2-flan": {
          "caption": "A solar panel sits on top of a railing on a balcony overlooking a city skyline.",
          "time": 41
        },
        "llama32-vision-11b": {
          "caption": "A solar panel on top of a table with a city skyline in the background.",
          "time": 43
        },
        "llava-13b": {
          "caption": "A laptop on a table with an urban skyline in the background, suggesting a workspace or study area in a city environment.",
          "time": 8
        },
        "llava-34b": {
          "caption": "A solar panel charging device on a rooftop with city skyline background.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "A solar panel on a rooftop with a cityscape in the background.",
          "time": 38
        }
      }
    },
    {
      "image": "./test-images/image-12.jpg",
      "captions": {
        "vit-gpt2": {
          "caption": "A man standing on top of a boat next to another man holding a surfboard in one hand and a surfboard in the other.",
          "time": 11
        },
        "git": {
          "caption": "Two men are in a boat, one of them is wearing an orange hat and the other is wearing an orange hat.",
          "time": 21
        },
        "blip": {
          "caption": "There are two men riding on the back of a boat in the water, one of them is on a surfboard and the other is on a board.",
          "time": 8
        },
        "blip2-opt": {
          "caption": "Three young men sitting on the back of a boat.",
          "time": 34
        },
        "blip2-flan": {
          "caption": "A group of people sitting on a boat watching a man ride a surfboard in the middle of the water.",
          "time": 46
        },
        "llama32-vision-11b": {
          "caption": "Two shirtless men on a boat watching people wakeboarding or surfing behind it.",
          "time": 43
        },
        "llava-13b": {
          "caption": "Two shirtless men on a boat, watching another man surfing in the water.",
          "time": 8
        },
        "llava-34b": {
          "caption": "Two shirtless men on a boat, watching another person water skiing.",
          "time": 16
        },
        "llama32-vision-90b": {
          "caption": "Two men on a boat watching another man wakeboarding.",
          "time": 41
        }
      }
    }
  ]
}