The Nebulous Nature of Software Engineering in an ML/AI Organisation

neb·u·lous /ˈnebjʊləs/ adj.: Hazy, indistinct, lacking clear definition; still taking shape. An apt word for software engineering within machine learning: a discipline inventing its own practices in real time, where the right processes may not yet have names.

Update (2026): Since this piece was written, Cord has rebranded to Encord and grown into one of the more closely watched AI-infrastructure startups out of London. In February 2026 it raised a $60M Series C led by Wellington Management, taking its total funding to roughly $110M at a reported $550M valuation. The interview below reflects the company as it was in its earlier, scrappier days.

The recent developments in machine learning (ML) over the past years have encouraged massive widespread interest, within technology teams and organisations of all sizes, to integrate intelligent ML and AI capabilities into their own products and services. This pushes organisations to evolve and adapt their existing development processes to this new and inherently experimental discipline. This article follows John Smith, a Software Engineer at a highly successful ML-focused startup, to gain an insight into the processes and practices these engineers follow within an ML-heavy environment. We look into the most common problems these engineers face, and how they resolve them. ML-focused teams at Microsoft, a very well-established company, possess a structure and set of processes differing quite greatly to that of an ML-focused startup. However, there is no one-size-fits-all solution for an ML team; there are several factors an organisation must consider before finding their ‘perfect’ ML structure and workflow.

Introduction

In recent decades, there have been countless disruptive technologies that have changed the way in which the world functions, from cloud computing to microservices to augmented reality. Behind the scenes, each of these developments has prompted necessary changes in the Software Engineering practices within a variety of companies. Machine Learning (ML), arguably the most impactful field in recent years, has evolved to be the preferred method for creating valuable systems for Computer Vision, Natural Language Processing (NLP), and several other applications¹. ML has benefited countless industries, from healthcare to marketing and sales, yet this advancement does not come without its several technical and organisational costs.

Software Engineering, at a high level, is the art of automating a task by writing rules for a computer to follow². A Software Engineer develops an application by breaking down a task into multiple sub-problems, solves these problems, and ensures that all of the solutions they have developed are entirely correct. This process becomes more intricate in an ML environment, since developing an ML application adds a considerable amount of uncertainty and variability to the development team’s workflow. Software Engineering processes must be completely reconsidered since the development, deployment and maintenance of ML-based applications differ completely from that of traditional software³. Advancements in ML have caused a shift away from the traditional view of software development (in which algorithms are written step-by-step using human expertise) towards ML systems which learn from data. As a result, teams and organisations must re-evaluate their existing engineering techniques and practices when designing and developing systems, to take into account the unique requirements of ML applications⁴. The development and deployment of ML systems is relatively fast and cost-effective today, yet maintaining them over time is difficult and expensive due to the additional technical debt specific to ML systems⁵. ML systems exhibit all of the issues of non-ML software systems, plus an additional set of ML-specific issues too.

Andrew Ng, a co-founder of Google Brain, compares the current rise of AI and ML engineering to the exciting inception of Software Engineering practices seen many years ago, stating: “The rise of Software Engineering required inventing [new] processes”⁶ and how today, “many AI teams use new processes that haven’t been formalised or named yet, ranging from how we write [documentation] to how we version data and ML pipelines”⁷. We are living in an exciting time for ML, where principles and practices that could be utilised for lifetimes are being developed and refined in these very moments.

The operationalisation and standardisation of clear-cut software development practices are crucial for the cost-effective development of reliable ML systems¹, yet, due to how ML applications are relatively new, several engineering practices for ML systems are still being developed and formalised. There is a stark contrast between how well established development practices are in stand-alone Software Engineering compared to Software Engineering within an ML environment.

The experimental and unpredictable nature of ML requires development teams to adjust their organisational structure as well as their technical practices accordingly. The magnitude of the change required, however, strongly depends on the size of the organisation, how established it is, and their level of focus and investment into ML. In this article, we speak to John, a Software Engineer at Cord, an early-stage ML/AI startup, to gain insight into their organisational structure and development practices. We focus on the ways in which an ML environment affects organisational structure, collaboration, the handling of development and operational issues, and technical debt. We compare our findings against ML-based teams at Microsoft.

Background

Cord is an early-stage ML and AI-focused startup, founded in 2020, which has already raised over $12.5 million in series A funding. At the time of writing, it is a small company with around 20 employees in total.

Microsoft is a very well-established organisation which effectively utilises ML in almost all product teams and areas. This is thanks to the mature Microsoft Research (MSR) division which, remarkably, around “20 years [ago], came up with the idea of applying [ML] to Microsoft products”⁸.

Organisational Structure

It is undeniable that ML has expanded significantly in the last few years. Yet as new as ML is as a science, the art of structuring an ML-based organisation or team is even newer. Interestingly, many current ML managers were pushed into leadership positions at companies simply out of necessity, with several coming from purely academic backgrounds⁹. In contrast, in several other organisations, many engineering or product leaders are involved in the development of novel ML features yet possess no prior ML experience. Leadership often doesn’t understand ML⁹, and there doesn’t exist just one correct way to structure an ML-based organisation. ML-based companies can be broadly divided into five archetypes¹⁰:

At Cord, each individual member of the development team is assigned an individual feature to develop. There is not always necessarily an ML component to these features, so it is not unusual to have one person following an entire feature or feature change all the way through (in what team topology literature calls a stream-aligned manner) without the help of any sort of enabling team. The exception is when a feature change requires the actual ML model to be integrated, which can only be provided by the ML team once the model has been fully developed and tested. In these cases, we can imagine a stream-aligned team (typically one person working on a single feature) with an enabling team stepping in at the very end. Platform and ‘complicated-subsystem’ teams are typically only seen in larger organisations¹¹ with a larger number of specialists, an example being Microsoft, where teams are able to utilise various in-house infrastructure and platform features ‘as-a-service’, allowing for a significantly increased rate of delivery¹².

As per Conway’s Law, Cord’s system structure clearly reflects their organisational structure: there exists one development team, with engineers individually working on separate features for Cord’s core platform, and one ML team, where models are developed and research into “what’s next” is being done. When these two subsystems interact, the two teams must communicate. There is also a vague split between frontend and backend development at Cord, a split that shifts depending on who is developing what.

Cord’s organisational chart as of March 2022 (names redacted). The chart shows the structure of the entire company (fewer than 20 employees in total), divided into ‘Engineering’ and ‘ML Research’. Software Engineers typically work on both frontend and backend depending on the task assigned; ML and Algorithm Engineers focus on researching and developing models.

Roles and ML/Non-ML Collaboration

John explained that, even in a very ML-focused startup, there is “a lot of work to do which is completely unrelated [to ML]”. As a Software Engineer at Cord, John never really has to do any “under-the-hood work with ML” and does “not need to talk to the ML [team] much at all”. His role focuses mainly on non-ML work such as building APIs, working with databases, and developing front-end UIs. The only, very infrequent, times when the ML team needs to contact a member of the development team is when a model needs to be integrated into Cord’s main platform, once a feature is pretty much ready to release.

Mature organisations, such as Microsoft and Amazon, would typically fall under the ‘ML-first’ archetype, where ML teams collaborate very closely with engineering teams. In particular, Microsoft tightly couples ML and non-ML development sprints and stand-ups¹³, which ensures close collaboration between the two teams, as expected in an ML-first organisation. However, this sort of structure is not typically found in startups¹⁴. Contrary to what the ML-first archetype suggests for startups, Cord better matches the ‘R&D ML’ archetype due to the lack of collaboration between the two disciplines. This may seem unusual, yet John feels it is actually better that way: “It would be very bad to have teams which are tightly coupled, since they do so many things which are completely independent of each other”. The engineering team are constantly “building features, maintaining the codebase, paying off technical debt, etc.” with fixed goals and strict deadlines, whereas the ML team are doing research with much more open-ended goals and without any strict deadlines.

This methodology is certainly cheaper, and hence more common in startups. On the contrary, having different individuals specialise in varying aspects of the project requirements will ensure high-quality work that adheres to best practices¹⁴. Cord has been extremely successful in their approach so far, and their team believes they can continue on this path without facing any major issues for now: “There is not much demand at all [regarding] mixing the two teams whatsoever; it wouldn’t make any sense at the moment”.

Technical Debt in ML

Technical debt is an especially unique problem in ML-based companies. ML systems hold a peculiar ability to suffer from this issue¹⁵, with ML being dubbed the “highest interest credit card of technical debt”¹⁶. ML offers powerful toolkits to develop complex systems quickly, yet it can be very dangerous to assume these ‘quick wins’ come without a cost¹⁶.

Generally, early-stage startups can actually benefit from taking on technical debt in order to accomplish short-term objectives. This temporary compromise allows for quicker iterations to gather more user feedback, and to scale (or fail) more quickly. Of course, this approach doesn’t come without its risks. In order to avoid hindering long-term success, these risks should be identified and mitigated.

John discussed the several factors affecting the relationship between code quality, rate of iteration, and technical debt within Cord. In the early days of the company, formal development practices, testing, and documentation were effectively non-existent. Although this may sound chaotic, John stated this did not hinder productivity; it was actually a major catalyst for Cord’s success. Without time-consuming and somewhat restrictive development practices, the founders were able to focus on simply building products that customers need, quickly and effectively.

Initially, only a couple of developers worked on Cord’s codebase from its very inception. These early-stage developers had a comprehensive understanding of the entire codebase, making a polished test suite and thorough documentation extremely low priority. However, as Cord’s development team grew and their product became more sophisticated, the absence of development practices began to hamper productivity. Newly-hired developers found it quite difficult to make sense of the codebase. As the number of developers increased, so did the amount of code, making it increasingly difficult for anyone to keep track of everything. To pay off the accumulated tech debt, developers are now building Cord’s very first test suite and set of documentation for existing features, on top of their responsibilities of developing new features requested by clients.

In addition to traditional technical debt, ML systems suffer from a variety of ML-specific problems. The common methods used to pay off technical debt within a codebase are simply insufficient here, as these ML issues exist at a system-level (affected by data) rather than solely at a code-level¹⁵. This makes ML technical debt exceptionally more difficult to both detect and mitigate. One example: on top of the standard dependencies found throughout software engineering code, ML systems typically contain several dependencies on various large data sources, which are usually more unstable than developers realise¹⁷.

Operations in ML

To quantify the maturity of a team or company’s ML production environment, Microsoft developed a Machine Learning operations (MLOps) maturity model¹⁸. This aids in clarifying the necessary DevOps principles and practices required to run a successful MLOps environment, ranging from Level 0 (‘No MLOps’) to Level 4 (‘Full MLOps Automated Retraining’). Interestingly, Cord has just recently shifted from Level 0 to Level 1 (‘DevOps no MLOps’). What characterises both levels is how the Engineering and ML teams are individually “siloed” without regular communication between each other. What pushes Cord to Level 1 has been the recent development of basic integration tests for the model, as well as unit tests for the application code. This maturity level is expected since Cord is still a relatively new company (especially when compared to Microsoft), yet has shown exceptional growth.

John described how there’s “no [single] DevOps person at the company”, and explained that Cord follows a ‘fully-embedded’ topology for their development and operations, also known as ‘NoOps’¹⁹, where operations are embedded entirely within the product development team. This topology is also used at Netflix and Facebook²⁰. This makes sense since all of these companies have one central product or platform. It would not be very useful for a company without a narrow product focus, since the financial and context-switching constraints present in an organisation with multiple product streams typically push the ‘Dev’ and ‘Ops’ components further apart²⁰.

Microsoft is very different in this context. Several years ago, many teams at Microsoft relied on development triads consisting of: a program manager (requirements and scheduling), a developer (programming), and a tester²¹. The adoption of DevOps by these teams merged the roles of developer and tester together, while IT, operations, and diagnostics were merged into the core software team¹³.

Since Cord is a startup, they utilise cloud services such as Google Cloud Platform (GCP) and Firebase for both their frontend and backend. Startups greatly benefit from the ease at which cloud technology scales as the company grows²². GCP provides Cord with both applications targeted towards developers requiring little ML experience, and applications targeted towards ML practitioners²³, a great fit for a team where Software Engineers are not very involved with ML. The cloud also makes it easy to experiment with ML capabilities and scale as features go into production and demand increases²⁴. Microsoft, as expected, typically utilises Azure and its many ML capabilities²⁵.

Cord’s system architecture, as sketched by John from memory and redrawn digitally. The frontend server handles user storage and login, communicating with a back-end API server and PostgreSQL database on GCP via HTTP, and a GCP-hosted message broker via WebSockets.

Conclusion

Having a significant ML component (whether in a team or an entire organisation) adds considerable complexity to the typical Software Engineering workflow. This is expected: developing capacity for a system to learn inherently adds a level of uncertainty into the system¹. An organisation or team with a strong focus on ML must understand several attributes before committing to a specific topology or workflow, including the maturity of the company, the resources available, and how tightly ML and product engineering need to interoperate.

Although the structure and operations of Cord differ greatly from those of Microsoft, both have very successful ML-based teams. As Cord grows, their structure and systems will adjust accordingly. The engineering processes and techniques which seem perfect today may not seem so in the near future.

Written by me together with two of my Imperial College London classmates, Guji and Amelia. With thanks to John for sharing his insights across multiple interviews, and to Dr Robert Chatley for his guidance throughout the module.