Demand for data engineers is growing rapidly and showing no signs of slowing down. In fact, DICE recently named data engineering as the fastest growing tech occupation in a 2020 report. But what exactly does a data engineer do, and what sort of pathways exist to become one?
Data engineers use a combination of programming and problem solving to design and build systems for collecting, storing and analysing data at scale. The goal of a data engineer is to process raw data into a form that can be used to derive valuable insights for businesses. Data engineers play a crucial role because without them, there can be no business intelligence, insights, or competitive advantage from data.
Data engineering is rarely an entry level role because it requires a combination of skills which are not often packaged into qualifications or training courses, unlike other roles such as data science. Most data engineers work in a related field like software engineering or data analysis before piecing together the skills to pivot into data engineering. This in my view contributes to the skill shortage, but it also means there are many pathways you can take to enter the field.
My pathway into data engineering was definitely not linear. I studied biomedical engineering at university and took a graduate role as a product development engineer at a medical device company. Although I enjoyed aspects of the role, I worried about becoming pigeonholed into a highly specific field so early in my career. I craved a steeper learning curve and greater flexibility to pivot and explore different options. That was when I turned to a career in data.
I started out as a data analytics consultant at a Big Four consultancy and quickly fell in love with the process of wrangling messy, shapeless datasets into tidy ones that contain actionable insights. I worked for a couple of years at the business intelligence end of the spectrum, before moving upstream to the data engineering systems and technologies which make the analytics possible. My shift from analytics to data engineering was motivated by a desire to understand the end to end data ecosystem. I wanted to understand the processes that took place before the data arrived at the dashboards and I wanted to have visibility of the end to end solution.
The transition to data engineering required not only honing my skills in popular data engineering tooling (primarily Python and SQL), but also upskilling in cloud technologies, which I did by studying for professional cloud certifications for AWS and Google Cloud. These certifications provided me with a good theoretical foundation which I could supplement with hands-on practice. They also taught me more general cloud engineering concepts which are crucial to grasp as a data engineer working in cloud environments.
I have worked across both AWS and Google Cloud, however I recently joined Kasna to focus purely on data engineering on Google Cloud. Google Cloud in my view is the best cloud for advanced analytics and machine learning workloads and has excellent integration between its big data services. They also have Looker which is an enterprise scale data platform for business intelligence. Looker combines slick data visualisation with a powerful data modelling layer.
If you are wondering if data engineering (on Google Cloud or otherwise) is the right career pathway for you, here are my reasons why I enjoy it and why you might too:
You never stop learning
Data engineering techniques and cloud technologies are evolving rapidly and every week there are new offerings from cloud vendors and third party tools being announced. Continuous learning and development is guaranteed!
You solve tricky problems
Data engineering projects are complex. You are often dealing with a combination of legacy systems and new technologies across hybrid or multi-cloud environments which bring many dependencies and challenges. If you enjoy problem solving, then you won’t be disappointed.
You make things better
Data engineers enjoy the satisfaction of building solutions which not only create genuine business impact, but also make the lives and jobs of end business users easier and more efficient.
If you are wondering if Kasna is the right place to build your career, here are my reasons why I think yes and why you might too:
Kasna lives and breathes Google
Kasna specialises exclusively in Google Cloud technology and is Australia’s premier Google solutions partner. At Kasna we get to work on cutting edge data engineering technologies and there are excellent people to work with and learn from.
Kasna is part of Mantel Group
Kasna belongs to a group of companies called Mantel group, with each company specialising in a different area of IT. Mantel Group provides amazing opportunities to grow your career at one or more companies depending on your goals.
Kasna nurtures diverse talent
Mantel Group is committed to tackling the diversity crisis in tech and has a number of early talent programs to encourage people into tech careers, particularly for women and gender minority groups.
Kasna is a great place to work
No, really. Mantel Group was ranked #1 in Australia’s Best Place to Work List.