Open LLM Leaderboard: DROP deep dive

Dec 2, 2023 ·

The recent Leaderboard Drop & Dive event hosted by Hugging Face was an experiment designed to run a leaderboard challenge on a popular language model. The challenge was to train a challenging dataset of up to 1,000 words in less than 16 hours using the HuggingFace Transformers library.

The participants were given access to two training datasets and two test datasets. The first was a classic task of question answering, and the second was a sentiment analysis task. Both tasks had to be completed within the 16-hour time frame.

The competition saw an incredible amount of participation from teams all around the world, with more than 500 teams submitting entries for each task. The winner of both tasks was team 'Jungle', who managed to achieve an accuracy of 96% in the Question Answering task and an accuracy of 86% in the Sentiment Analysis task.

The competition also saw participants creating innovative new ways of solving the tasks, such as fine-tuning their models and leveraging external data sources. This demonstrates how language models can be used to solve complex problems and highlights the potential for using them in real-world applications.

Overall, the Leaderboard Drop & Dive event was a great success, with Hugging Face receiving positive feedback for its efforts in organising the challenge. It was a hugely exciting experience for all involved, with teams competing to find the best solutions to the two tasks and to push the boundaries of what is possible with language models.