VeloxCon 2024, the premier developer convention that’s devoted to the Velox open-source mission, introduced collectively business leaders, engineers, and fanatics to discover the newest developments and collaborative efforts shaping the way forward for information administration. Hosted by IBM® in partnership with Meta, VeloxCon showcased the newest innovation in Velox together with mission roadmap, Prestissimo (Presto-on-Velox), Gluten (Spark-on-Velox), {hardware} acceleration, and way more.
An outline of Velox
Velox is a unified execution engine that’s constructed and open-sourced by Meta, geared toward accelerating information administration programs and streamlining their improvement. One of many largest advantages of Velox is that it consolidates and unifies information administration programs so that you don’t must hold rewriting the engine. As we speak Velox is in varied levels of integration with a number of information programs together with Presto (Prestissimo), Spark (Gluten), PyTorch (TorchArrow), and Apache Arrow. You possibly can learn extra about why Velox was inbuilt Meta’s engineering weblog.
Velox at IBM
Presto is the engine for watsonx.information, IBM’s open information lakehouse platform. Over the past yr, we’ve been working arduous on advancing Velox for Presto – Prestissimo – at IBM. Presto Java employees are being changed by a C++ course of primarily based on Velox. We now have a number of committers to the Prestissimo mission and proceed to accomplice intently with Meta as we work on constructing Presto 2.0.
A few of the key advantages of Prestissimo embrace:
- Hugh efficiency increase: question processing might be performed with a lot smaller clusters
- No efficiency cliffs: no Java processes, JVM, or rubbish collections, as reminiscence arbitration improves effectivity
- Simpler to construct and function at scale: Velox provides you reusable and extensible primitives throughout information engines (like Spark)
This yr, we plan to do much more with Prestissimo together with:
- The Iceberg reader
- Manufacturing readiness (metrics assortment with Prometheus)
- New Velox system implementation
- TPC-DS benchmark runs
VeloxCon 2024
We labored intently with Meta to prepare VeloxCon 2024, and it was a unbelievable neighborhood occasion. We heard audio system from Meta, IBM, Pinterest, Intel, Microsoft, and others share what they’re engaged on and their imaginative and prescient for Velox over two dynamic days.
Day 1 highlights
The convention kicked off with periods from Meta together with Amit Purohit reaffirming Meta’s dedication to open supply and neighborhood collaboration. Pedro Pedreira, alongside Manos Karpathiotakis and Deblina Gupta, delved into the idea of composability in information administration, showcasing Velox’s versatility and its alignment with Arrow.
Amit Dutta of Meta explored Prestissimo’s batch effectivity at Meta, shedding gentle on the developments made in optimizing information processing workflows. Remus Lazar, VP Knowledge & AI Software program at IBM introduced Velox’s journey inside IBM and imaginative and prescient for its future. Aditi Pandit of IBM adopted with insights into Prestissimo’s integration at IBM, highlighting function enhancements and future plans.
The afternoon periods have been equally insightful, with Jimmy Lu of Meta unveiling the newest optimizations and options in Velox. Whereas Binwei Yang of Intel mentioned the combination of Velox with the Apache Gluten mission, emphasizing its international affect. Engineers from Pinterest and Microsoft shared their experiences of unlocking information question efficiency through the use of Velox and Gluten, showcasing tangible efficiency beneficial properties.
The day concluded with periods from Meta on Velox’s reminiscence administration by Xiaoxuan Meng and a glimpse into the brand new easy aggregation perform interface that was introduced by Wei He.
Day 2 highlights
The second day started with a keynote from Orri Erling, co-creator of Velox. He shared insights into Velox Wave and Accelerators, showcasing its potential for acceleration. Krishna Maheshwari from NeuroBlade highlighted their collaboration with the Velox neighborhood, introducing NeuroBlade’s SPU (SQL Processing Unit) and its transformative affect on Velox’s computational velocity and effectivity.
Sergei Lewis from Rivos explored the potential of offloading work to accelerators to reinforce Velox’s pipeline efficiency. William Malpica and Amin Aramoon from Voltron Knowledge launched Theseus, a composable, scalable, distributed information analytics engine, utilizing Velox as a CPU backend.
Yoav Helfman from Meta unveiled Nimble, a cutting-edge columnar file format that’s designed to reinforce information storage and retrieval. Pedro Pedreira and Sridhar Anumandla from Meta elaborated on Velox’s new technical governance mannequin, emphasizing its significance in guiding the mission’s improvement sustainability.
The day additionally featured periods on Velox’s I/O optimizations by Deepak Majeti from IBM, methods for safeguarding towards Out-Of-Reminiscence (OOM) kills by Vikram Joshi from ComputeAI, and a hands-on demo on debugging Velox functions by Deepak Majeti.
What’s subsequent with Velox
VeloxCon 2024 was a testomony to the colourful ecosystem surrounding the Velox mission, showcasing groundbreaking improvements and fostering collaboration amongst business leaders and builders alike. The convention supplied attendees with worthwhile insights, sensible information, and networking alternatives, solidifying Velox’s place as a number one open supply mission within the information administration ecosystem.
Should you’re eager about studying extra and becoming a member of the Velox neighborhood, listed below are some sources to get began:
Keep tuned for extra updates and developments from the Velox neighborhood, as we proceed to push the boundaries of information administration and speed up innovation collectively.
Strive Presto with a free trial of watsonx.information
Was this text useful?
SureNo
