The future of software engineers

Creating this topic to understand the view of fellow engineers. I tried looking for a proper category but didn’t find where to put this. In case I can move this to another category, please suggest a category where there would be an broader audience.

I was going through all the introductions for the upcoming Part 1 2020. I felt a bit overwhelmed by the backgrounds of all the people . I did my undergrad in Electronics & Comm. engineering, but was into software dev. immediately after college and have been doing this stuff for the past 5 years.

So when seeing architects/doctors/artists/pathologists etc enter ML/DL, I think they would do a better job in understanding the problem from their respective fields than that of a simple software engineer like me can. They certainly have the background to do that.

So I would like to know your thoughts as to what would be the role of software engineers in the upcoming years in ML/DL when experts from the each field start working on their own problems. I would love seeing Jeremy/Rachel talking on this topic.

Will keep adding my questions/thoughts here.

7 Likes

I’d say they still have a role with building these models, as not every artist, biologist, etc will pick up ML. So I see it as a collaboration where we build the models and they help us to understand the flaws, and give context to the behavior of the models and if it’s “good enough.” Certainly the ones that convert over will be able to do this too, but I know for a few of my projects (where I am a CS major), my partner is a herpatologist (snakes) and so we consult ideas back and forth and I make adjustments on our models.

3 Likes

I think this doesn’t change current situation. Software developers will keep building great tools so specialists from other fields could easily solve their problems using these tools.
Fastai, or pytorch, or TF are examples of such tools in ML/DL. There will be always CS and software engineering problems to solve.

4 Likes

Also the same issue is already true for traditional software development. Domain experts can create more appropriate software in their domains than software engineers, since they understand the needs and constraints better.

In general, when creating a solution for a domain, you should work very closely with folks that live and breathe that domain! Regardless of whether it’s an ML or s/w eng solution.

20 Likes

The answer from Jeremy is so good I am considering tattooing it on my forearm!!! So good! Something that would serve me well to remember!

I would like to share a related story from another community I find very exciting, that is Ruby on Rails. When RoR came out over 10 years ago now, it was said that they eliminated the role of a DBA thanks to the design choices made in the framework. Working on an agile team nowadays it is hard to even imagine (unless it is a very specific piece of software) having someone there whose primary skills would be databases.

10 years into the evolution of the framework and apparently the word on the street is that the role of backend engineer is being cut! So lucky of me to switch to ML as that is what I was doing before :smile: Jokes aside, this is a big thing, backend engineering has been the core of many teams. But the tools evolved to the point where it no longer requires for this hardcore expertise to be embodied in a single person or a couple of people on a team. Just look at all the amazing tooling available out there for spinning up APIs.

If we trace a line through all this, look at the big picture, technical competencies are becoming less relevant and the user experience and ultimately domain knowledge, which has always been crucial, receives even more spotlight.

5 Likes

Reading the responses, I was wondering if colleges should kind of restructure their courses around ML(assuming basic programming being already taught) and the other domains (Law/medicine/bio tech/genetics/astronomy and what not. Fast.ai has professionals coming in almost every possible domain). Though programming in general kind of touches all the fields, but now with ML, I think this is definitely going to be common subject taught at college. The same image classification is working to classify songs(sounds) and dogs. The same techniques but a wide range of applications across different fields.

I was interested in Bio Technology when I was entering my undergrad. But then, in India, the career opportunities were less when research is not an option or at least I was informed that way. But now if we combine the programming skills(specifically ML) and the domain knowledge of Bio Tech/Bio Med, it would be wonderful :slight_smile:

4 Likes

In my opinion, as one progresses in their software eng. career, one of the most useful and transferable skills between various organizations/products/teams etc. is the ability to take the time and try to understand the domain in question as thoroughly as they can. Every other technical skill (which, is undoubtedly extremely important) comes secondary to this.

This is, also how a team/engineer would produce comparatively better software. It probably isn’t too far fetched to say that, a team that understands a problem/domain better than any other, and well equipped with professionals(not only eng., usually some domain experts included) will easily beat other solutions.

From my experience, some products/domains don’t require a lot of upfront heavy learning, while some others do. Take a classifieds website for an example, the domain is not extremely difficult to learn, so most of the focus of eng. team is on user experience + perf. optimization. On the other hand, for a fintech startup, you’d have to start digging into how banks work, borrowing/settling, dunning periods, market regulations, how money moves around etc., before you even write any software, which of course will require all the usual eng. practices. This is probably also why some businesses are harder replicate than others.

Software engineers seem to be in need for the time being, so there’s probably no reason to sweat on that. :crossed_fingers: But, one would be a much valuable professional if they trained themselves to be good+quick at understanding domains they’re not really familiar with, and then translate it into ‘machine instructions for automation’. (This is also probably a very good reason to work a company/product that you care about.)

Once again, please don’t get me wrong, all the technical skills will definitely be required and they all add up, but getting into this mentality of bridging/translating business domains and user needs into software design/requirements will prove to be an even more valuable skill in the long run. (among many more others)

Lifelong learning is really the strongest strategy, most likely common in many other professions as well.

5 Likes

I don’t think software engineers are going away anytime soon (or at least I hope not!)

Part of it is that I think it will still be rare in the future to see someone who has gained expert-level knowledge in a domain and in software/ML. That’s just hard to do, and even as the tools get better and easier, it will still take considerable personal investment. So it will still be efficient from a time and knowledge perspective for a doctor who can code to collaborate with a software and/or ML engineer with an interest in and even some knowledge of medicine/biology.

And that’s aside from the fact that I think there will actually be an increasing demand for people working on ML infrastructure, tools, and libraries (just look at the growth of PyTorch and fastai in the last few years), which involves a lot of “pure” software work along with some understanding of what specific domains need (even general libraries can and should benefit from the input of domain experts).

2 Likes

Building the infrastructure to roll out ML is also not trivial. The gap gets smaller all the time but there’s still a gap between running something in your notebooks and integrating it into an organization that doesn’t stand still while you do it.

Thanks to projects like fast.ai I’m fairly confident that the model/get something to run and produce pretty good results part is more or less solved. I’m fine (grateful would be more fitting) with plugging in SOTA models tuning them to my data and getting good results.

As programmers, we are really good at one thing and that is automation. For me data is the key to good ML and data comes in strange formats from different sources. That’s where we can shine in the future.

I’m not worried, I’m pretty excited :slight_smile:

3 Likes

:muscle:

1 Like

This creates a very interesting topic in itself-I have very strong opinions about a Uni degree or studying a concept there.

I finally found the best way to articulate the issue during an interview, to quote the guest:

“The issue is most of the academicians who teach courses have never been Data Scientists or ML Engineers so their courses might even lead you back into teaching it, but wouldn’t be the best to prepare you for the real world”.

I’m not saying there is any easy way to solve the problem-you need to have experience to be experienced, that’s why fast.ai family is a huge propeller of “doing” rather than “just studying”.

4 Likes

I’d think both domain experts and software engineers will co-exist. The pie is simply that big. Although anyone who can do both will be highly coveted in the industry. We can compare it to building websites today. It has gotten trivial but still too much of investment and digression for non-tech people to get themselves involved in.

2 Likes

Making predictions about the future is hard. Here is a documentary about Artificial Intelligence from 1983. Expert Systems were all the rage back then and everyone thought people would be out of jobs by the turn of the century.

Curiously enough, there is one mention of an “artificial neural network” amidst all the excited talk about the rule based systems that were popular back then. That particular neural network was actually just a jumble of wires but it is shown to recognize faces. I’m not quite sure how they trained it back then. I believe this was before Hinton and Yan LeCun even came on the scene.

Having said all that, I think the role of Software engineers may change but it’s going to be around in the near future (next couple of decades.)

This paper “Software Developers Learning Machine Learning: Motivations, Hurdles, and Desires" is a very interesting read and won best paper award at IEEE.

In this paper, we sought to understand: a) the motivations of
software developers learning to adopt ML, b) the technical
and knowledge hurdles they encounter, and c) their specific
desires for how modern ML frameworks can better support their self learning needs.

8 Likes

Google drive is blocked for me (in my office). Found this alternate source http://pgbovine.net/publications/software-developers-learning-machine-learning_VLHCC-2019.pdf (in case its helpful for any one else)

It’s worth bearing in mind that once you’ve worked on ML in a particular area, particularly when you’ve debugged a lot of models, you may find that you inadvertently become an expert in that field! Obviously this isn’t going to happen overnight, but if you spend a few years in the same area then it will start to happen.

2 Likes