New Research Reveals 97% of Data Teams Are at or Over Capacity

PALO ALTO, Calif., July 23, 2020 /PRNewswire/ -- Ascend.io, the data engineering company, today announced results from a new research study about the work conditions of data scientists, data engineers, and enterprise architects in the U.S. Conducted in June 2020, findings from over 300 professionals reveal key insights on their teams' current workload, productivity bottlenecks, and perspectives on automation and low-code technologies. Since the onset of the COVID-19 pandemic, the majority (78%) of data professionals have been asked to take on responsibilities outside of their core job function, with 97% now signaling their teams are at or over capacity. To support their teams and increase overall capacity, 89% of data professionals are turning to automation, low-code, or no-code technologies, with 73% citing automation as an opportunity for career advancement.

Pressures on Data Teams Escalate
The overwhelming majority of data teams are currently at or over work capacity, with just 3% citing they have extra capacity for new projects. Moreover, data executives (including VPs and directors) are nearly two times more likely to indicate their teams are overloaded than team leads and individual contributors themselves, signaling a significant backlog and increasing demand by the business. When asked which team was the most backlogged, enterprise architects led the way, followed by data engineers. However, respondents were over 3.5 times more likely to identify their own team as the most backlogged over others. Despite pointing to their own team as the backlog, the research found compelling patterns emerging around the need for data engineering resources to complete their work.

All Roads Lead to Data Engineering
Slow iteration cycles in data teams lead to significant delays for downstream users and decision makers, hindering teams' ability to meet the data needs of the business. The delays vary by role: data scientists are mostly impacted by having to ask others for access to data or systems (48%), whereas data engineers are mostly held back by maintenance of existing and legacy systems (54%).

"Organizations are quickly discovering that data engineers are essential to unlocking the value of data and to removing bottlenecks across the entire data team," said Sean Knapp, CEO and founder of Ascend.io. "LinkedIn's 2020 Emerging Jobs Report found that data engineering has surged onto the scene, quickly becoming one of the top-ten jobs experiencing tremendous growth. At present, there are simply not enough data engineers to meet the demand. To enable more data professionals to tackle the growing backlog of data engineering tasks, tools such as automation, low-, and no-code technology can provide tremendous leverage and scalability for existing data engineers, while at the same time enabling a new era of citizen data engineers."

Data Teams Level Up With Automation to Advance Careers and Empower Teams
Data professionals are increasingly turning to automated solutions to solve their team's current challenges. 62% of respondents plan to implement automation technology to increase their team's bandwidth, followed by buying new products or tools (47%), replatforming and retiring legacy technologies (41%), and hiring more staff (41%).

The majority (73%) of respondents view automation as an opportunity to advance their career.

Enterprise architects are the most welcoming of automation, with 85% citing that it would provide an opportunity to further advance their career, followed by data engineers at 72%.

Perspectives about what is holding data teams back are more diverse depending on who was asked:

• 77% of financial services respondents plan to implement automation to increase their team's bandwidth, compared to the overall average of 62%. The more competitive nature of some sectors is perhaps driving faster adoption of data technology to drive business results than in others.
• Meanwhile, 75% of healthcare respondents indicated plans to replatform and retire legacy technologies, compared to the overall average of 41%. Some industries may still need to modernize their overall technology stack before data teams can have a greater impact on the business.

More Than Half of Data Engineers Seek Low- and No-Code Technologies
The survey results indicate that across the spectrum of startups to large enterprises, data teams today are looking to low- and no-code solutions to accelerate innovation and productivity. More data professionals are seeing the value of these tools, with 80% of respondents already using or considering them to support their team.

25% of data engineers are using low- and no-code solutions today and 53% are currently considering such technology, compared to 48% of enterprise architects and 42% of data scientists.

"The research demonstrates that across the board, teams are being asked to do more with less and they are turning to low-code solutions as a means of continuing to deliver on business needs," continued Knapp. "Low-code solutions are great at simplifying what used to be incredibly complex tasks. They also tend to encourage better modularity and system design, which keeps things simpler and more maintainable."

Additional Resources 

• Visit the Ascend.io Data Engineering category page
• Download the research study infographic
• Follow Ascend.io on LinkedIn
• Follow the Ascend blog

About Ascend.io
Ascend.io, the data engineering company, provides the only unified platform for data pipelines with 10x faster build velocity and automated maintenance. Data pipelines are the backbone of modern data systems and require a modern approach to meet the exploding demand for new data initiatives. Evolving beyond traditional ETL and data orchestration tools, the Ascend Unified Data Engineering Platform makes it possible for teams to easily build autonomous data pipelines that dynamically adapt to changes in data, code, and environment. In a radical departure from existing open-source and commercial solutions that require excessive coding and don't operationally scale, data engineers using Ascend can now ingest, build, integrate, run, and govern advanced data pipelines with 95% less code. Ascend's DataAware™ intelligence uniquely observes, understands, and tracks every piece of data, enabling data pipelines to run at optimal efficiency with integrated lineage tracking, auditability, and governance. Freed from maintaining brittle pipelines, data engineering teams can now invest in bringing the growing backlog of business ideas and opportunities to life. Ascend empowers more people to become citizen data engineers and helps teams derive insights with a faster "time to why," accelerating the innovation journey from prototype to production. The Ascend platform can be deployed on top of existing Apache Spark clusters or as a fully managed solution.

Media Contact
Andrew Smith
Bhava Communications for Ascend.io
[email protected]
(310) 941-7251

SOURCE Ascend.io