The model was submitted under the offline scenario in MLPerf's open division in the BERT 99.9 category. The goal was to maximize throughput while keeping the accuracy within a 0.1% margin of error from the baseline, which is 90.874 F1 (SQUAD). The DeciBERT-Large model far exceeded these goals, reaching a throughput of 116 QueriesPer Second (QPS) and an F1 score of 91.08 for accuracy.
For the submission, Deci leveraged its proprietary automated Neural Architecture Construction technology (AutoNAC) engine to generate a new model architecture tailored for the AMD processor. AutoNAC, an algorithmic optimization engine generating best-in-class deep learning model architectures for any task, data set, and inference hardware, typically powers up to a 5X increase in inference performance with comparable or higher accuracy relative to state-of-the-art neural models.
"While the key optimization objective when generating the DeciBERT model was to optimize throughput, AutoNAC also managed to significantly reduce the model size – an important accomplishment with a number of benefits including the ability to run multiple models on the same server and better utilize cache memory," said Prof. Ran El-Yaniv, Deci's chief scientist and co-founder. "These results confirm once again the exceptional performance of our AutoNAC technology, which is applicable to nearly any deep learning domain and inference hardware".
MLPerf gathers expert deep learning leaders to build fair and useful benchmarks for measuring training and inference performance of ML hardware, software, and services.
The Impact of Faster NLP Inference
Deci's NLP inference acceleration directly translates into cloud cost reduction as it enables more processes to run on the same machine in less time or alternatively it enables teams to use a more cost efficient machine while retaining the same throughput performance. For some NLP applications such as question answering, higher throughput also means better user experience as the queries are processed faster and insights can be generated in real time.
Deci Submission Results
|
Hardware |
F1 Accuracy on SQUAD (INT8) |
Model Size (in Million parameters) |
Throughput (QPS) ONNX
Runtime FP32 |
Throughput (QPS) ONNX
Runtime INT8 |
Deci's
Boost |
BERT Large |
Dell-PowerEdge-R7525-2xAMD-EPYC-7773X |
90.067 |
340 |
12 |
18 |
|
DeciBERT Large |
Dell-PowerEdge-R7525-2xAMD-EPYC-7773X |
91.08 |
115 |
76 |
116 |
6.64x |
About Deci
Deci enables deep learning to live up to its true potential by using AI to build better AI. With the company's deep learning development platform, AI developers can build, optimize, and deploy faster and more accurate models for any environment including cloud, edge, and mobile, allowing them to revolutionize industries with innovative products. The platform is powered by Deci's proprietary automated Neural Architecture Construction technology (AutoNAC), which empowers data scientists to build best-in-class deep learning models that are tailored for any task, data set and target inference hardware. Leading AI teams use Deci to accelerate inference performance, enable new use cases on limited hardware, shorten development cycles and reduce computing costs. Founded by Yonatan Geifman, PhD, Jonathan Elial, and Professor Ran El-Yaniv, Deci's team of deep learning engineers and scientists are dedicated to eliminating production-related bottlenecks across the AI lifecycle.
Media Contact
Garrett Krivicich, Headline Media
[email protected]
+1 786 233 7684
Photo - https://mma.prnewswire.com/media/1893621/Deci_Throughput_Infographic.jpg
SOURCE Deci
Share this article