The future of Data Engineering. Is this role going to vanish? Tech Recruiters must also read
I’m damn sure, if you already have a job as a Data Engineer, you can’t know everything in this image above. Well, that’s precisely my point regarding the Data Engineering role. It is not reasonable to master your job.
Today, to be a Data Engineer, you must know the following:
- Data Analytics (ITIL is a must)
- Python for automation (3 years experience is a must)
- Create and manage Data Pipelines (not just Airflow, come on)
- Cloud Environments (I mean, every single infrastructure solution for Data on every cloud)
- Infrastructure as Code (Terraform)
- Oracle Database, SQL, MySQL, NoSQL, EverySQLonEarth;
- DBT
- Spark
- DataBricks
- Kafka (I can do this all day long, you know that)
So, as far as I know, the Data Engineer needs also to be a full-stack developer. Am I Right?? I think I made my point here.
The trend I’m following
The discussion I see is that even super seniors (10~20 yrs experienced) people are frustrated or confused about this. I know a guy who was interviewed for a Data Engineer role and the reason for being denied was Django!
On the other hand:
I can’t imagine why vacancies… layoffs? Don’t think so :)
Look how focused the companies are to hire Data Engineers:
I hope tech recruiters can read this: it is not hard to find the Data Engineers with your dream requirements, it is impossible!
How to fix this?
Of course, I have the solution, and it is easy! Stop overwhelming these people!
The Data Engineer Role must be split into 4:
- Data Analytics Engineer
- Platform Engineer
- DataOps Engineer
- Database Reliability Engineer
Data Analytics Engineer
- Data Analytics
- Python (Pipeline creation and management)
- Databricks
- SQL
- Spark
Platform Engineer
- Infrastructure as Code (Terraform, Bicep, Ansible, Puppet, etc.)
- Cloud Environments
DataOps Engineer
- Kafka
- DBT
- How to make everything works
Database Reliability Engineer
- Oracle Database, SQL, MySQL, NoSQL, EverySQLonEarth;
- On-premise expertise
- Hybrid environments expertise
Achievements
If you are a Data Engineer, you can now focus on what you are best at and try the roles that already exist (yes some companies have been doing a great job looking into the future).
If you are a Tech Recruiter, you can now understand why is so hard to hire Data Engineers. Now it is time to focus on learning with your tech team leaders what they need.
If your company got used to hiring a super senior that knows everything, I got 2 messages:
- There is no such thing as Full-stack Data Engineer.
- You need to hire specific people for specific jobs if you want deep knowledge.
I hope you enjoyed this reading, follow me for more, and have a nice day!
And by the way, what do you feel about all this? Share here your thoughts!
