Turing Provides Vision & Language Model Evaluation Technical Support for “Heron VLM Leaderboard”- In Collaboration with Weights & Biases Japan, Provider of the ML Experiment Management Platform –

Turing Inc. (Shinagawa-ku, Tokyo; CEO: Issei Yamamoto; hereinafter “Turing”), a company dedicated to utilizing generative AI to develop fully autonomous electric vehicles, announces that they have provided technical support for the “Heron VLM Leaderboard,” launched today by Weights & Biases Japan (hereinafter “W&B Japan”), a provider of ML experiment management platforms. At launch, the leaderboard allows for comparisons of more than 15 VLMs through the Japanese VLM evaluator “Heron-Bench.”

Overview of “Heron VLM Leaderboard”

The Heron VLM Leaderboard combines the expertise of Turing, who is developing the multimodal generative AI “Heron” for fully autonomous driving, and W&B Japan, which operates the “Nejumi LLM Leaderboard,” the largest LLM leaderboard in Japan. Integrating Turing’s proprietary Japanese VLM performance evaluation benchmark “Heron Bench” with the globally-recognized VLM evaluator “LLaVA Bench (In-the-Wild)” provides a comprehensive evaluation that assesses complex real-world images within the cultural context of Japan.
URL: http://vlm.nejumi.ai/

Interactive Analysis of Model Evaluation Results

The Heron VLM Leaderboard leverages the experimental management functions of the Weights & Biases platform to ensure evaluation reproducibility and traceability, providing interactive displays and an analysis of evaluation results. The leaderboard captures the characteristics of each model by displaying average scores in a ranking format, offering radar charts for comparing performance across multiple models on multiple evaluation axes and tables, and displaying the input and output of each evaluation task.

About “Heron Bench”

“Heron Bench” is a versatile benchmark developed by Turing to evaluate Japanese VLMs. It consists of pairs of images and questions specific to Japan split into three categories: Conversation, Detail, and Complex. This process provides a detailed analysis of Japanese VLM performance.
URL: https://huggingface.co/datasets/turing-motors/Japanese-Heron-Bench

About Turing

Turing is a startup dedicated to developing fully autonomous vehicles. The company is developing an end-to-end autonomous driving AI that makes all driving decisions based solely on data obtained from cameras. Turing’s goal is to achieve fully autonomous driving by 2030 through the development of “Generative AI” for high-level autonomous driving, “Semiconductors” for on-board AI operations, and “Autonomous Driving Systems” that integrate vehicles and AI.

Company Overview

Company Name: Turing Inc.
CEO: Issei Yamamoto
Established: August 2021
Capital: 30 million yen (as of September 2022)
Headquarters: 4th Floor, East Tower, Gate City Osaki, 1-11-2 Osaki, Shinagawa-ku, Tokyo
Business: Development of fully autonomous vehicles
URL　　：https://tur.ing

Recruitment Information

Turing is seeking colleagues to change the world through fully autonomous driving. We frequently host company introduction events and autonomous driving experiences, so please feel free to contact us.
Recruitment page：https://tur.ing/jobs

Media Contact

Public Relations (Abe)：pr@turing-motors.com