Succeeding at scale: How large teams can implement CI/CD and move faster

Share with

Description

As team size increases, some challenges can be harder to solve. Join Richard Owen, Senior Product Manager at Gearset, as he talks through how larger teams can navigate these challenges by implementing CI/CD to move faster and succeed in DevOps.

Topics discussed:

  • Why is Salesforce DevOps challenging?
  • How large teams can succeed at scale
  • How to measure success and review your own processes

Learn more:

Transcript

So, yeah, thanks very much, Tiffany. I'm really excited to be talking to you all today about succeeding at scale, how to help large teams implement CICD successfully and move faster.

So first, a quick introduction. I'm Richard. I've been a senior product manager at Gearset for almost two years now, and prior to this, I worked for ten years in financial services, making marketing credit risk software for enterprise sized teams. And my focus at Gearset is functionality which helps teams, especially large teams, work better together.

As part of this, I've been the product manager in charge of the pipelines functionality for the past eighteen months or so. I've been working with about one hundred teams of different sizes to help make their day to day processes easier to manage.

And today I want to share with you what I've learned about how to help teams and especially larger teams be successful as they adopt CICD processes.

So here's what I'm going to be talking about today. First, I'll explore some of the reasons why adopting DevOps processes in Salesforce can be challenging. Then we'll look at have a look at the difference team size makes and why this will become more of a challenge as team size increases.

After that, I'll share some of the tips that we've used to help teams succeed when adopting those processes, along with some examples of tooling and functionality we've developed, which is helping a bunch of teams today. And then at the end, we'll have a very quick look at next steps or what teams should do when they get to a good state.

So with that, let's get started.

So the big question, why is Salesforce DevOps challenging? So at a high level, there are three main factors to this. So first, DevOps processes are primarily about organizing people in your team, and this can be really quite a tricky thing to do on the Salesforce platform because, as you all know, there's such a spectrum of technical backgrounds across most teams between the no code admins configured on one side and pro code developers on the other extreme. And there are also technical release managers, business users, other roles who have slightly different requirements and needs, and this technical diversity can pose quite a significant challenge.

Secondly, the Salesforce platform is traditionally focused around clicks, not code, and the developer centric features have grown around that a lot slower than has been the case in more traditional software platforms. And Salesforce also comes with some idiosyncrasies, some quirks, which means that you really need a bespoke set of tools for the job.

And thirdly, whilst DevOps is definitely not a new concept, it's been around for more than a decade and much longer than that in terms of the underlying practices themselves, it is still fairly new to the Salesforce ecosystem.

As a result, some of the cultural aspects of DevOps and the working practices are they can still be relatively unfamiliar to a lot of teams working with Salesforce.

So let's start with the technical technical diversity challenge. So on a big picture level, the DevOps process being a clear process for developing, deploying, and shipping changes has to be inclusive. It's got to work for the whole team or you're gonna end up with silos and you'll end up bottlenecks on the least robust part of your process.

So if you've got some silo team members who are still making changes directly in production, they're deploying org to org, or they're manually back syncing changes between orgs, then you're still gonna have slow manual releases, and you're going to tread on each other's toes all the time.

And this we can represent this by a spectrum of these kind of technical backgrounds, and I'm sure some of you will be able to picture where you sit on this. At one end, you've got the real the folks who really live by the clicks not code mantra. At the other end, you've got the more traditional pro developer, and this can lead to a split experience really easily across the platform where different people have different and diverse approaches to creating changes.

So users on your team may have a widely different level of familiarity with source control and get in the overall level process, and different groups across the team will end up working independently. They'll end up working in silos. And this can have a really quite bad consequence, which is that you end up lacking a common, clear, and externalized release process.

The second issue, that we've highlighted here is around technological barriers on the Salesforce platform. So Salesforce is a fantastic user driven platform, but, traditionally, it hadn't been built for collaboration.

It's quite often difficult to see who's working on which items and features. It's really common to step on each of the on other people's toes and overwrite other users' changes, for example, if you're changing and deploying layouts or profiles.

And this means that it's quite easy for things like merge conflicts to arise.

And, finally, there are also complex dependencies in metadata and some idiosyncrasies in the API, which have to be overcome as well to be effectively able to work with it.

And finally, on the on the cultural side, as I said before, DevOps is not new. It's been around in general software for over a decade because it works.

Teams using these processes work better together, and they move faster.

One reason teams adopt DevOps processes is to improve communication and break down silos. If you've got development ops teams and release managers who don't have good lines of communication between them, it introduces delays in your process.

So when this is improved, teams release faster, and they release more reliably.

And this works best when you've got lightweight in team approvals, and you're moving as much of the process as early as possible, shifting left, I think, is the term, to raise issues early so that they don't come as a surprise on release day, and you've got much better knowledge across the team of what's happening at any given time.

So, essentially, we're looking for a workflow which moves from these separate silos. Everybody's working in their own area with poor communication between them to a much more inclusive workflow where everybody's involved in the process, everyone knows where stories have progressed to, and how their work interact with the work of others.

And on top of that, you've got much better and closer communication across the team.

So these challenges are far from being unique for large teams. They exist in teams of all sizes implementing a robust and sustainable DevOps process on Salesforce, but team size does make a difference. As team size increases, these challenges can bite you a lot more.

So for a team of thirty, making sure that they can communicate effectively so that users know what their colleagues are working on is much, much harder than it is for a team of three.

For a start, if for a team of thirty, they'll almost always, always in my experience, be organized into multiple logical teams in the business. A single sprint team won't contain thirty people. So this immediately introduces silos built in.

As a result, there are gonna be changes made by users on their user stories, which others on the team are not aware of. And on Salesforce, that's even more of a challenge. There are so many global kind of cross cutting objects, things like profiles, record types, layouts. There are there are many of those metadata types, which can impact otherwise unrelated stories being worked on by others in the team.

As a result of it, conflicts that you see in Git involving those objects will come up a lot more frequently as team size grows, and that's even before you consider genuine conflicts between objects being impacted in metadata or Apex.

There's another challenge that comes in as team size increases. There's a a division of roles that tends to become more pronounced.

So I've worked with a lot of three or four person teams where everybody has responsibility for their changes from right from the the start of the process all the way through to production, but I've seen very few twenty person teams for whom the same is true. And by that scale, teams usually have a dedicated release manager or architect who manages the process of promoting changes through to the little the later environments, du UAT, pre prod production, with developers being responsible for their changes at the early part of the process.

And for for those larger teams, that larger volume of changes being pushed through can also lead to changes in the release schedule. So most teams that I work with with more than ten members tend to release on a set cadence rather than pushing features to production on demand. And there can be many factors which play into this. The desire end users know exactly what's being released on a predictable schedule comes up quite often. They need to have approval by a change approval board before changes are pushed or just plain inertia in the team. That's the way it's always worked.

So how can teams of this size, often spread across multiple sprint teams of the company, be successful in implementing a DevOps process?

So this is a strategy that we've put together here, and it's it will help teams of all sizes set up for success. It applies across the board, but it becomes ever more important as team size grows.

But first, know where you're starting from. Assess your starting point. Know who you who you want to bring into the process, what your roles are, and where they're happiest working. So some users will be living in Versus code and version control. Others prefer a GUI to understand and push changes, or they need that top down view of the whole process because you need to know where all of the changes are with at a glance.

Secondly, don't move too far too fast. If you've got nothing in version control and you're using chainsets or just starting with auto or deployments, moving straight to a full fledged release pipeline with version control being source of truth in one shot is gonna be really, really hard to do successfully.

Define the steps that you want to take in the long term and plan out to do it step by step.

Thirdly, start adopting those key tenets of DevOps culture, transparency, good communication, and shared ownership processes.

So that's the shifting left part. That allows errors to be caught really early in the process and encourages teams to keep all of their environments in alignment as much as possible, reducing noise as you're pushing items through.

Then once you've got to that great that great place, it's really versatile not to stop. So elite teams are always iterating on the process. They're measuring improvements so that they can understand how they should be keeping improving over time.

And understanding that lead time for changes, the success rate for deployments, and return on investment, those key door metrics that Jack and Rob were talking about earlier is critical for long term success.

So as a product manager for the Gizet pipelines, I've worked with, I said, about hundred teams at different stages of their journeys looking to get the release process up and running. So as we go through these stages, I wanna also refer to some of the tips, the best practice recommendations, and share some of the functionality that we've built as well at Gear Sets, which helps those teams succeed.

So let's start with the the starting point. To make the best decisions, you need to know where your team's starting from. And we usually do this through the DevOps and Charity Spectrum. So here's a view which you've, possibly seen on our on our website. It's going through the different stages from novice through to expert.

And as teams move along that spectrum from left to right, their process enables faster deployments, shorter lead times, and lower failure rates. And if things do go wrong, they can recover far, far more easily from this. But, of course, from this, teams can't tell exactly from this picture where they sit because, you know, whether their team is a bike, a car, or a rocket.

Right? So we need to quantify what each of these actually means. And for that, we can turn back to the Dora metrics.

So, quite busy slides. Apologies for that, but, this shows the kind of thing which you would expect from teams at different stages of that maturity spectrum. And these there are data metrics developed by Google to monitor the success of their DevOps processes. And we'll be hearing more about this again in, Charles' presentation coming up after this.

As teams move from chainsets to Octo deployments, they start adopting version control and then rely on it as a source of truth. We see the cycle of releases shortening significantly.

Deployments which may have taken weeks of planning previously can be done on demand several times a day. The The time taken for a story to go from the initial PR into production can be measured in hours, recovery times in minutes, and the success rate of deployments as we see from our own stats and gear set goes well in a well in excess of ninety percent.

But as I said earlier, importantly, if your team's still at the initial stages of adopting DevOps processes, trying to move from the left hand side to the right hand side in a single shot is unlikely to be a smooth journey, and we'll talk about that more in step two.

So the best thing to do there is to figure out where your starting point is, who you need to be involved in the process, and how they want to work, who needs to be who needs to be working with a GUI, who wants to work in version control and Versus Code to decide how you want to start the process.

And this is where we see this is the case where we do see the pipelines UI coming in. So here's a screenshot from that on the left hand side and a view of, this in Git as well. So, admins or users less familiar with Git workflow terminology will traditional will will normally be living in the GUI, which effectively translates that functionality to steps they need to take to progression item, whilst developers may prefer to deal with PRs and Git or checking directly in Versus Code. And pipelines becomes a kind of Rosetta Stone for those cases. It helps all users of the team have a common view of what's happening at different stages of the process from those first commits, those first check ins all the way through to production.

And we go into the second step, start small and scale up. So one of the most stark stats that we've seen over the three years that we've run the state of Salesforce DevOps reports is that just nine percent of elite teams adopted DevOps with a big bang approach.

That's really important to realize that that big bang approach, in other words, launching DevOps in a single step, has a huge risk of failure associated with it because there are so many variables, with adopting that process. So it's better to use an incremental iterative approach to implement DevOps, making sure they're making those steps and then measuring that success as you go. And that enables the organization to focus on continual improvements and get buy in across the company as well and make sure that all groups are collaborating every step of the way.

Another aspect of the start small scale up, as in when we mentioned, we were moving through the DevOps maturity scale. We had having version control as your source of truth as the pinnacle. That's the ultimate goal. That's the point to aim for. So let's get everything into VCS and start pushing changes through.

No.

That's generally not the way that we find, the teams are finding success here because Salesforce metadata structure means that some metadata types are simply not suitable for keeping under version control, and others, like dashboards and reports, tend to be environment specific. You don't want your testing dashboard to be pushed straight through into production.

So choose the metadata you want to version carefully, test out changes through CI jobs on continuous integration. And what we typically see is that teams who get this up and running well at the start use version control for about twenty five to fifty metadata types, and this covers the vast majority of changes they'll ever wanna be pushing through.

We also recommend pushing changes through using Delta CI or commit by commit deployment. And that way, you can be confident that the only changes going to the org are the ones that you've committed and the ones that you actually want to be pushing, and that keeps your deployments small and fast.

In terms of environments as well, they need to bring all of your environments under the sun into a pipeline or into a release process in a single shot. Adding new environments, branches, and sandboxes is easy later on. So you can start with your critical path, your integration environment, UAT production, and then as you're ready to bring other environments in to add complexity to it, you can then bring in some more dev environments, hotfix environments. You can build up those branches as you go once you've got that initial process well established.

There are also some cases where we wanna start small and not scale up, and that applies to feature branches and Apex. So we wanna keep feature branches as small as possible as it minimizes the chance of chance of hitting merge conflicts throughout the release process.

And for many metadata types, including layouts, record types, we support deploying only the parts of those objects which have changed, which we call precision deployments, just as we've supported breaking down profiles into their components for several years. And this makes it a lot easier to avoid conflicts and just deploy deploy the parts of those cross cutting objects which are needed to complete the story that you're working on.

And as I mentioned there, Apex is another case where it's really good practice to reduce the size of classes being used Because the smaller the classes, the less likely it is that multiple users are gonna be hitting the same objects.

And then moving on to possibly possibly the most important stage of this, which is adopting principles of dev ops dev ops culture and shifting left. So this is summed up nicely in the quotes there. If it hurts, do it more often. Bring the pain forward. If you're able to test early, get code in those first environments under source control, and understand the conflicts and issues that that come up as early as possible, the chance of having items fail late in the development cycle is dramatically reduced.

There are many aspects of how Salesforce teams can achieve this from improving visibility of changes across the team, dealing with merge conflicts early, keeping environments better in sync, through to getting good integration with ALM tools like Jira to be able to expose results out to the business better, and define the release schedule which will work for your team. And these are some of the primary areas that I've been working on with our pipelines functionality to help teams adopt those best practices. So in this section, I'd like to share some of this tooling and workflows that we've enabled for those teams as well as some of the pitfalls that we've seen teams encounter over the past couple of years and some of the things which have slowed slowed some teams down in their adoption process.

So first, visibility across the team. So going back to that pipelines UI view, it's really important for everyone in the team to have a common view of the state of the world, and that's where the pipelines UI comes in from. So it shows a high level visualization of all of those changes in flights, including merge status, conflict resolution, validation, work item tracking, that real kind of bird's eye view of of the state of the world for your team across all of your environments. And it gives you that view of where all of the stories are at all parts of the process and is accessible to all members of the team.

On top of that so now I wanna talk about merge conflicts. So merge conflicts, they're they're not an inherently bad thing. They stop you from overwriting somebody else's work without asking and make sure that you're working constructively together. And it's critically important here to flag up those genuine merge conflicts so that they can be resolved early in the process. So that way, users can talk with each other early and resolve them before they become a much bigger problem later on when you get closer to production.

And, yeah, pipelines UI will show that shows that clearly, exposes the conflicts, and allows you to choose how to resolve them. Of course, a bunch of conflicts that you normally get in Git are down to Git not understanding the structure of XML, particularly for some of those cross cutting objects.

And for those, the semantic mode algorithm that we have inside pipelines has knowledge of the Salesforce metadata structure and allows you to successfully merge most of those, reducing the amount of noise that you have to deal with here.

Another key factor is environment alignment across the pipeline, and this can hit you in two different ways.

So a key a key cause of problems in your release process, if you're looking at, your production environment, UAT integration, any earlier environments, is environment drift. If there are many changes in master or your production environment which aren't in your lower environment, so changes which are being made directly in production, new features will often hit conflicts or validation errors when you try to merge them in as you go through the process.

So what we see is that what we recommend is the teams merge back from master into their lower environment frequently. It makes the whole process cleaner and ensures that your environments are much better aligned. So some of the cases where I've seen teams get into real trouble is if they let those environments drift further and further out of alignment because every time it does, it increases the noise, increases it increases the friction process as they go through subsequent releases. And the same is true the other way as well. If there are lots of changes which have been abandoned over a long period of time, earlier environments like integration UAT can become overgrown with lots and lots of changes which never make it to production.

And each of those changes is a new difference. It's a new little kind of landmine or a source of different source of drift between environments, which has the potential to cause conflicts further down the road.

And what's worse is that in those conflicts, it becomes more and more likely that you need to resolve them in a different way on different environments, and that makes the drift worse than it already is. So our recommendation there is to identify those features, revert them out, and get back to a state where the environments are better aligned with production.

In addition to that, integration with, ALM tools like Jira or Azure DevOps work items allows information about changes to be surfaced to the wider business more easily.

And, it's nothing that we see in pipelines where we can push information about promotions happening to the tickets themselves with links to the PRs, and we also can allow, the status to be updated automatically as features make it into key environments. And This is something which we've seen really help unblock teams. Because if you know that a feature is ready for testing, you can pick it up without needing a manual prompt from your teammate and becomes this becomes a lot more crucial as team size increases.

And then finally, when it comes to releasing the production, know what you want to achieve. Know what you're aiming for in terms of a release cadence. So different teams have different requirements for this step. I know we've said that feature by feature releases further along the DevOps maturity scale, but it's not always viable, especially for larger teams. And this can be for many reasons for, that I've mentioned earlier. Users wanting to know when changes are gonna hit their production environment with certainty, the requirements of business processes, or just team's preference to work to release out of hours to cause minimum disruption and avoid locking the production environment.

So you can choose to either do the feature by feature deployment or you can build up a release over time, validate it continuously against production, and then deploy that when you're ready to go. So that functionality in our pipeline is being used today by dozens of teams to deploy on a set cadence. And individual features can also be deployed as hotfixes, and that makes sure that you don't have to wait for release dates to push out really, really urgent stuff.

So overall, these process changes, how do these changes impact teams? So from the state of Salesforce step ups report there, when team members have a clear understanding of the process to get changes improved approved for implementation, this just drives high performance across the board. Increased productivity, improved release quality, more frequent releases, reduced lead times. It pushes everything to the right.

And, the stat which came up in the previous presentation, ninety eight percent of teams reporting seeing, return on investment from adoption of Salesforce DevOps practices, which is quite unbelievable.

So, yeah, what's next? So with all these processes in place, with the release cycle shortened, features moving into production quicker than you've ever done before with a higher rate of success, what's what's the next step?

So teams of any size, but particularly those larger teams, can't rest on their laurels. They've gotta be reassessing and iterating on their process to ensure what efficiency. And this is reflected in this quote from, from Bill Holtz, Gardner, that successful DevOps organizations embrace continuous learning. The most important characteristic of high performing teams is they are never satisfied.

They always strive to get better.

And with this, there are several areas teams can focus on. Taking tasks which are manual today and continuing to automate them really removes and reduces operational risk in the process.

Make you're making sure that you're testing what you have to test at each stage, back up back up to those strategically important logs regularly, and monitor those changes to ensure people aren't circumventing the process.

Because catching those unexpected changes in people who are working outside the process earlier is most of the battle to make sure that you stay in a good state.

And then finally, keep measuring as you go because how can you tell the impact of any change that you make when you're not measuring it? And the benchmark for this, as we saw earlier, is those Dora metrics covering release met release frequency, lead time, success, and recovery rates. And our reporting API goes a long way to understanding this data.

So from that reporting API, we can use that to track performance, output into dashboards, and expose it to management about the business, and this can also be embedded within your own Salesforce instance. So that makes it really easily accessible across the business.

So what's the message that I wanna leave you with today? I I know it's a bit of a cliche, but this is so, so true in the subject. It is a marathon, not a sprint. Know what you want to achieve and move there step by step. You'll see the improvements in process and the reduction in time taken to push changes through as you go. And keeping the team happy with these changes is absolutely critical as having the team working together with these best practices makes the whole process a lot more streamlined, and all these steps becoming become more and more important the larger the team becomes.

And with that, I wanna say thanks very much for listening today. Here are some links, so please get in touch with any questions that you have and links by find if you wanna give it a try. And with that, I will hand back over to Tiffany.