Although tech companies are racing to build bigger and better artificial intelligence models, there’s still significant value in smaller models. Microsoft is doubling down on that concept.
Microsoft on Tuesday launched Phi-3 Mini, the first of three small models the company says it’ll launch in the coming months. Microsoft trained Phi-3 Mini on 3.8 billion parameters, or variables that AI models use to deliver better results. Phi-3 Mini is the smallest of the three models Microsoft plans to launch. The company didn’t say exactly when to expect Phi-3 Small, which will have been trained on 7 billion parameters, or Phi-3 Medium, which will have been trained on 14 billion parameters.
To put these parameter numbers into perspective, some reports have suggested that OpenAI’s GPT-4 Turbo was trained on more than 1 trillion parameters. Last week, Meta said that when its final Llama 3 model launches later in 2024, it will have been trained on 700 billion parameters.
Also: Microsoft unveils Phi-2, a small language model that packs power
The more parameters a model is trained on, the more capable it is of delivering the kinds of results users would want, but this comes at a cost. The more parameters an AI model has, the more power and energy it requires to deliver results. While more parameters may be best for complicated queries or mission-critical AI implementations, like those in health care, that’s not always the case.
Indeed, smaller models like those Microsoft is developing are great for smartphones and other lower-powered devices. Microsoft could use Phi-3 in mobile devices, where on-device AI performance is constrained by chipset power and battery life.
Despite its smaller size, Phi-3 Mini performs well, Microsoft claims. In an interview with The Verge, the company said that Phi-3 Mini offers the same performance as models trained on more than 10 times the number of parameters Microsoft used, and although it can’t match GPT-4 or GPT-4 Turbo, it is as capable as GPT-3.5.
Microsoft told The Verge that the company trained Phi-3 Mini on a “curriculum” that included children’s books to achieve that performance. The company also used a larger model to craft AI-generated children’s books to supplement its real world material.
Microsoft is making Phi-3 Mini available for free on its Azure cloud platform, model collaboration site Hugging Face, and AI model service Ollama.