Schoolboy created website where AI competes in Minecraft building

Traditional AI benchmarking methods have proven to be insufficient, so AI developers are turning to more creative ways to evaluate the capabilities of generative models. One group of developers decided to use the popular sandbox game Minecraft for this purpose.

TechCrunch writes about it.

Why Minecraft is used for benchmarks

The MC-Bench website was developed to compete with artificial intelligence models in a face-to-face confrontation by responding to prompts with Minecraft creations. Users can vote for which model did the best job, and then they can see which AI created each building.

The platform was founded by 12-year-old Adi Singh, and for him, the value of the game lies not only in the game itself but also in the fact that people are familiar with it. Even those who haven't played the game but have heard about it can appreciate which block representation of pineapple is better.

Currently, the MC-Bench team consists of eight people who work on a volunteer basis. Singh says he wants to understand how far we have come since the GPT-3 era, and games can be a great environment to test the capabilities of AI models.

The MC-Bench benchmark explains the complex with simple values, rather than when we see benchmarks in which OpenAI GPT-4 scored 88% on the LSAT but cannot determine how many r's there are in the word strawberry.

As a reminder, OpenAI released the powerful AI model o1-pro in the API for developers, which made it the most expensive in the company's history. You need to spend at least USD 5 for the OpenAI API services, and the company asks for tokens starting at USD 150.

We also wrote that the entrepreneur from the Netherlands created the browser game with code written entirely by AI. In the first month, it brought him USD 90 thousand in revenue, despite the fact that he has no programming skills.