alt Hacker NewsIt's frustrating how the data discipline is split up into so many different job titles. I regularly have to work with data engineers, data analysts, analytics engineers, data scientists, machine learning scientists, and machine learning engineers. Each view their role really narrowly, so you need to pull in a lot of people to get something done. It's often easier to just do the work yourself.
I get that some specialisation is useful, but I would like to see more T-shaped data folks.
Replies
How scary deep end to end ownership of some kind of a data analytics pipeline solution is, depends on a lot on organizational maturity which based on what I've seen is not high when it comes to data. In-house developers are avoiding these kinds of responsibilities not due to a lack of skill or ambition but because your job just becomes "churn dashboard e2e pipeline expert". You somehow can not be a T shaped developer and use your broad expertise in hopping in and out of projects and domains.
Stakeholders prefer a single point of dev contact so you can never get rid of the role, and you can only repeat that for so many domains or solutions before you run out of hours in the day. Trying to spread this ownership across a team results into either horizontal or vertical specialization which are the two dysfunctions. I'm not sure why this is like this in data.
Agreed, after decades writing scala, go, rust and python and building high throughput analysis systems, I'm now not allowed to touch any of our infrastructure, or write any airflow jobs because our teams got partitioned into data engineering and data analysts, and I was given the analyst team.
30 min livebug fix and review for me on a job I specced out? Ticket to another team to review and schedule for next sprint and thus 2 weeks of metadata likely lost or malformed.
These setups look great (but full of invisible friction) when things are going well but shit the bed as soon as you come across the simplest of human errors that needs a solution.
Weird. I feel the other way. I do too many things from all areas that none of the titles fit me. I build pipelines, analyze data, engineer the infrastructure around data orchestration, etc.
It used to be the case that a "data engineer" was just a software engineer that was good at data-related things (maybe they were good at Kafka, airflow, DBT, knew why you'd pick postgres for one job and Clickhouse for another, etc).
Then the BI people (who had never written a line of code except maybe some SQL) got jealous, started calling themselves "data engineers", broke everything ("what do you mean 'we need tests???'"), and ruined the title.
My favorite title is 'digital prostitute' - we do anything if you pay us money :D
Could you describe a typical day-to-day thing that each of those roles does? (I see the terms often, but I couldn't say what each of them actually, ya know, does.) Thanks!
> It's often easier to just do the work yourself.
And yet if a data person were to try the same on your specialization, surely this would also be well and good.
Yeah, as a "T-shaded data folk" (have worked as a data analyst, data scientist, data engineer, and machine learning engineer, so I think I'm close to a full bingo care) in data, I agree!
I think the specialisms are really helpful for hiring - data engineering and data science involve a lot of different things so it's a lot easier to hire someone if you don't require skill in both.
That said, I think we have a culture problem in defining our roles narrowly, I've seen data scientists look down at data-pipeline parts of their job, and ignoring model latency and deployment; data engineers be deeply uncurious around what happens to the data after they put it in a warehouse; etc.
The best people I've worked with, within data and elsewhere, are the ones who are curious about every aspect of the stack, even if they mostly focus on one side.