Description
Catch up on this session with Aidan Harding (Technology Director, Nebula Consulting) from the Gearset DevOps Summit 2022. Aidan runs us through the challenges and opportunities of DevOps on Salesforce, covering such topics as:
- Why deploying on Salesforce isn’t always simple
- Learning Environments
- Is the only source of truth production?
- “Cargo-culting”
- As well as solutions, ideas and benefits of Salesforce DevOps.
Learn more:
Salesforce DevOps — What is it, and why’s everyone talking about it?
Transcript
You, everyone. Hello, DevOps nerds. Thank you to Gearset for inviting me here. I'm glad that Rob mentioned philosophy because that's, one of the topics here.
I wanna talk about the philosophy challenges and opportunities of DevOps and Salesforce. So pretty quick, slide about who I am. I am, Hayden Harding. I'm the, technology director at Nebula Consulting.
We are a consulting Salesforce partner, based in London.
And I think that gives us an interesting perspective on DevOps that a lot of the, stories that we hear from people are about end users and ISVs and things and how they use it.
But as a consultancy, we can't always control everything. We can't change the, culture of multinational companies that we work with every time. So we have a slightly different perspective on things, and we're establishing a different process, multiple times. Every time we come into a customer, we've gotta make decisions about what's the right approach for them. So, we have to really understand what it is we're trying to achieve, what they're trying to achieve, and try and make the right decisions. So a few other things on that, slide about myself, but lots more interesting things to talk about. So if we had to sum up, Salesforce DevOps in a single image, how we do that, bin fire in a flood.
If you've ever tried to deploy during a Salesforce release or you've tried to deploy experience cloud or any number of other scenarios, you will have some empathy for the bin fire. But I would at least emphasize that that bin is smiling as he goes through the flood. So it's not all bad. There are good things.
So what are we gonna talk about?
Simplicity. So deploying on Salesforce is not that simple, and, there are some good reasons for it. We're gonna talk about different kinds of learning environments and why it might be hard to improve things as we work on our, DevOps journey. We talk about source of truth. People like to say that, source control is the source of truth, but, I'm going to argue that the only source of truth is production.
We talk about cargo culting. If that's something that you're not familiar with, then don't worry. I'm gonna explain it. If it's something you are familiar with, don't worry. I've got nice pictures that will make it entertaining.
And we talk about some different approaches, to DevOps as well.
So first off, the illusion of simplicity.
Really, when it comes to deploying on Salesforce, all I wanna do is I just wanna move my code from a to b without drowning and pulling my own tears.
That's kind of what I've got in my head. But, that word just, particularly for consultancies, it could be a bit of a sort of triggering word. Often, customers would say, I just want this. I just want that. Meaning that they think it's simple, but maybe it's actually harder.
So to give an example of that, let's say somebody had a book that they just want to localize from US English to, British English, and they think that that's simple. All we have to do is we just replace search and replace all the words that are American sounding like pants, and we put in English sounding words, say, trousers.
That sounds fine, except now we've made a new word, particip trousers.
So this is an actual picture of an actual book that's been translated and, when the participant pants of the revolution became the participant trousers of the revolution.
And I think deploying code on Salesforce is a bit like that. You feel like it ought to be simple, but, actually, there's, real complexity to it, real challenges to it, particularly when we're working as a team. So, we actually do need to take a good approach to it. One of the other challenges that we have is the kind of learning environment that there is for, working on DevOps.
So I read an interesting book, called Range by David Epstein, which has been excellent. It has nothing to do with DevOps, but it has this concept of kind and wicked learning environments. So a kind learning environment is an environment where you get feedback on your act actions unambiguously and quickly after you take them. And that makes it easy to improve in that area by just doing the work, seeing what happens, and then learning from the results.
But there are a whole other type of environments, wicked environments, he calls them, where the feedback might be delayed or it might be unclear or there might be no feedback at all. And in those environments, it's really hard to improve what you're doing without taking real deliberate action, experimentation, reflection to see how you're doing.
And to put some examples of that, kind learning environments are things like sports, games, and in programming. We get linters and compilers give us instant feedback on how we're doing.
But the more wicked learning environments, things like parenting, parenting is pretty hard. You try and do this and that. You don't know, what result you're gonna get at the end. And DevOps is just like that. We introduce some changes to how we do things. We might not be able to tell how well that's worked for six or twelve months, so we might have to create reports and monitoring to actually find out how our changes are doing, whether they had the kind of effects that we want.
And Salesforce in itself is complex. The real world of deploying stuff in Salesforce hurts. There's real fundamental complexity to that. There's not accidental complexity where sometimes you introduce complexity just because of how you tried to solve a problem, which we would call accidental. But I think in Salesforce, there are fundamental reasons why it's quite hard sometimes.
One is that Salesforce sells itself on being easy to configure. So changes made in production from Salesforce's point of view are not a bug. They're a feature. That's why people bought the thing. They want to be able to create a new field, create a report, tweak a layout. That's why they went for Salesforce, not something else.
As Salesforce release new features, they often prioritize getting those features out there over making them deployable. You know, it looks great in a keynote. It looks good in a video. But can we deploy it properly yet? Probably not in the first iteration. So this creates extra complexity for us, things that have to happen outside of our deployment pipelines.
Often things that are really metadata that are configuring things, like, especially in packages, say CPQ has a whole load of records in the data part of the system that are really configuring stuff. And so when we build our, processes, we need to consider this metadata that's hiding as data.
Some deployments are not atomic, which makes life very difficult. So something like changing the type of a field, if that field is referenced somewhere else, might take multiple deployments to actually get there, which causes potential problems. If you get partway through and get stuck, then the system's in an inconsistent state.
Some particular types of things are hard to deploy. Communities, experiences, as they're called now, just don't deploy very well, and some metadata objects are just way too big. They have too many responsibilities. So profiles, page layouts, record types, all of these things are quite hard. If you've got multiple teams working in parallel, they all want to change these objects because they have too much responsibility.
And lastly, as I mentioned before, as a consultancy, we can't rely on getting buy in from all the stakeholders to make a wider cultural change, with a DevOps approach. Like, ideally, we would like to, but if we're a company of thirty people, we're working with a multibillion dollar multinational company, we're not gonna change the whole of how they work. We're just gonna have to try and make it work the best way we can for the teams we're working with.
And finally, my cat, who you may have seen just jump on me just now, would like to remind you, don't forget the data. When we think about deploying, we think about deploying flows, permission sets, Apex, but, also, that tends to come with data changes. So we need to consider data backups, rollback. If we have to rollback the whole deployment, we may need to rollback the data as well. There's always more to do with data than you think. I can't remind you of that.
So going back to this concept of the source of truth, I think production is the source of truth. I don't think we can reliably say that source control is. So one reason for that is just to share a number of metadata items. If you go to the metadata coverage report Salesforce, you'll see a whole number of metadata types and how, you can deploy them. So when I first gave this to work earlier this year, there was there were five hundred and forty eight.
In the latest release, five hundred and seventy seven. I looked earlier today. It's now five hundred and seventy eight. So this is very large number with a moving target of more and more metadata types. And from Nebula's point of view, we currently, sync twenty four metadata types in our CI system. So that's about four percent of the total. So at absolute best, we haven't, got everything in source control, to give a full pack picture of what's in production.
But there's more than that, the philosophy. Plato's cave. So, Plato had this allegory of a cave to try and talk about, the need for philosophy and the, inadequacy of your direct senses. So to talk you through this allegory, on the left hand side, we've got shadows on the cave wall, and we've got people sat on the floor looking at the shadows. And these people have were born in the cave, raised in the cave. All they've ever seen is shadows on the wall.
And so when they're seeing a bird there, they have no idea that a bird is the thing that can actually fly around and flap and, sing. All they know is the shadow on the wall. And so what they're seeing are these shadows. It's these people in the hooded robes bringing the objects across, and that is really their perception of what is going on.
But, really, over on the right hand side of the picture, we could see people escaping the cave to the real world where actual things are happening. And Plato's point was that, we shouldn't just trust our senses to understand the nature of the world. We need to use our minds to think philosophically about the nature of things and get beyond just what we see. But this isn't just a picture.
There are, real analogies to what we do in Salesforce. So if I put an actual shadow on the cable of an octocat, you might start to see where this is going.
The shadows that we see on the wall is the source control. The CI system is delivering those shadows to us, but, really, production lives above ground, and it's not the same as what we're seeing. And as poor developers, we're sat there, looking at the shadows, maybe crying a bit, and that's all we can see. And the real thing about this is that if somebody reports that something's gone wrong in production, we cannot fully trust the source control, the shadows on the cable to find out what's going on in production. It may be that, there's a difference between reality and the source control. We've always got to bear that in mind when we're diagnosing things. Is it something that's not quite, what we expected?
So is source control the source of truth? Well, yeah, but not completely is what I would say. It'll only ever contain a fraction of the metadata.
It can miss changes made directly in production, which is to say people probably should be able to make changes in production, certainly some changes.
There could be changes in source control but hadn't made it to production for some reason. Say some other admin operation has locked the org, so your CI system hasn't been able to put things in. There can be those discrepancies between the two two things.
So saying that source control is truth is an abstraction, but it's a leaky abstraction, and we've got to remember that it is just that, when we're thinking about things. So cargo cults.
The idea behind cargo cults is just the a warning that we shouldn't just try to copy the behaviors of large successful companies and bring them into what we're doing without understanding why they're doing it. So, one way to put in that is you are Google, unless you are. I guess there could be people here from Google. So, hi, if you're from Google. But if you're not, then you're not from Google, so don't just copy them.
And as this is there, the processes that we might copy are meaningless without understanding the intentions behind them.
The, the next part is a quote from the agile manifest manifesto in where they say that we should prioritize individuals and interactions over processes and tools. And if we just start copying the way Google do things without understanding why, then we've got that backwards. We start prioritizing the processes and tools over the individuals and interactions.
So it was nice to hear Rob, talking about DevOps processes in this way that they are actually a collection of things, and you choose what's appropriate for you. You don't have to take every single thing that a higher performing, DevOps company is doing and adopt them yourselves. You pick the things that are right for you, and you do the things that are right for you at the right time for you.
So where did this whole, phrase cargo cults come from? It sounds like a weird thing. The origin story is from World War two in Melanesia, whereas Melanesia, you may ask, is the sector of Pacific Island between New Guinea and, Fiji.
So at that time, American Air Force, had bases in these islands and that they were coming in with their planes and their electricity and their canned food and all this stuff that the islanders had never seen before. And, they were sharing them with the islanders, and, eventually, the Americans left. And the islanders were left wondering how could we continue to get these lovely things we were getting when the Americans were there. And they didn't really understand, the technology that the Americans had to get that stuff there.
So, what they did is they tried to repeat the things they'd seen the Americans doing in the hope that, that would bring the same benefits. So they copied the symbols. They created their own landing strips. They made, aircraft out of hay bales.
They made lights for those, landing strips out of fires, but, unsurprisingly, it didn't have the effect they wanted. And I'm not saying this to, look down on the islanders at all. This is exactly the same thing as we do as engineers. We sometimes get this level of, I wanna be like Google.
I wanna be like Facebook. So I'll just do what they do, and then we'll be big and successful like them.
But, things just don't follow in that way.
This is another, great haystack picture here, and it's a particularly good reminder, of the dangers of copying stuff around without understanding where it came from.
I love this picture, but if you look in the background, that house doesn't look like it's in, Melanesia. It looks quite English, and that fence also looks quite English. And where this picture actually comes from is, Nantwich in Cheshire in England, where an ice cream company, has a marketing promotion every year where they buy build, giant sculptures at haystacks.
And where I got this from was, from a talk about agile where somebody was talking about cargo culting had included this picture and not fit actually looked into where it come from. So in a way, they'd kind of cargo coated this picture around. So even people who know about cargo coating are still vulnerable to the influence of copying around things without understanding why.
But to bring this right back to, DevOps and the kind of things we were talking about, deployment frequency.
You know, from the, Dora metrics, we'd say we wanna be an elite team. We wanna be deploying multiple times per day.
What does that even mean? If you deploy multiple times per day, are you actually putting features in the hands of your users multiple times a day that they're actually gonna be able to use? Possibly. Possibly not. It might be appearing and available to them, but maybe they're not using it. Or maybe you're using feature flags to hold it. So you're incrementally deploying parts for feature for six months, but no one could actually use it till you switch that feature flag at the end.
And then in that sense, you're actually having one great big release when you switch that feature flag. It's probably better than trying to deploy six months work in one go in one massive deployment, but you're kind of half kidding yourself about doing, multiple deployments if that is all you're doing with them. And I think one of the, factors here is the good old Goodhart's Law that when a measure becomes a target, it ceases to be a good measure. And so if we suddenly start targeting multiple deployments per day rather than, and we're giving people that target, incentivizing them towards it, then they will optimize for just that target. Whereas if we're using it to measure to see how well we're going, then it becomes more useful. And if we understand why we want to to deploy multiple times a day to reduce the risks, to get features into the hands of users so we can get feedback, then we start to get more sensible stuff around this.
So that was all depressing. Everything's terrible. How about something more constructive? How about some actual approaches to, deploying stuff on Salesforce?
So something that I really like as a pulling method on Salesforce in a way of working in a team is unlock packaging. So from that, what we do with unlock packaging is we take a Salesforce org and we break it into, to, packages that we install like AppExchange packages, but we're writing them all ourselves and we're managing all of that. And we have more freedom than ISVs do in our unlock packages.
So here's an example where we've got some different packages. One to do with sending Massey on my own with the SendGrid, one to do with archiving data, with our invoicing system, with all these different things. And, we built them all as unlocked packaging, and your work can function as a whole with these. So let's look at the benefits and drawbacks to unlock packaging.
You get a clean separation of functional areas if you, use unlock packaging.
If I wanna change something to do with SendGrid, I just go to the SendGrid package. That's what I have to change. Everything's in there. Great.
It really is source driven. When I work on a monoc package, I check the whole thing out of source control.
I build a scratch org on the basis of that, and that is everything. There is nothing there's no part of it that isn't in source control.
That's great. You could do it with all Salesforce tooling. The CLI can manage all that for you, which is great. And org dependent unlock packages were a later addition, and they're really useful because they allow you to build unlock packages that don't force you to decompose all your existing code into packages before you can get started. And it means that parallel work on separate packages is really easy to do.
We're just working on separate code bases, and then we can, integrate and test them later on. But there are downsides.
Not everything is packable packable. Not every metadata item that you could deploy with CI is also, deployable with packages.
It's not great for customizing the standard clouds. If you're doing a lot of work around opportunities in cases, then you tend to run into stuff that just doesn't fit very well with packages. It works better with, custom objects.
If you do wanna decompose an existing code base into a a collection of unlocked packages, that's actually quite a hard engineering challenge, and it's quite hard to justify the effort to do that.
It can be hard to manage dependencies between packages. There's no version of NPM that can just, sort all your dependencies for you, which is a shame.
It might require dependency injection to break your, packages into separate pieces of code, and that's a relatively advanced dev topic. But you might well wanna do that anyway, because it allows teams to work separately even if they're not in separate packages.
The time to start working could be quite long with unlocked packaging. Creating and configuring a scratch work can take a while. But, overall, like, unlocked packaging, I think, is excellent. And if you can live with that list of caveats, I would always pick that as the cleanest way of doing it because it really is source driven, and we really can just work with Scratch Works. It's great.
So I talked about the source of truth.
How about we just say production is truth? And we roll with that.
Bill Woe Bill Woe Baggins can create it in prod, so he's gonna create it in prod. So let's work with Bill Burr. Let's not fight him. Let's find a way of doing that. So if we have a normal, kind of system where we think the source control is the source of truth, then we'll have a one way deployment. Every time someone does a commit, we push that into production there.
If Bilbo's around, then we should probably add a second arrow and allow a scheduled deployment back into source control to pick up all the changes that are happening directly in prod.
This does, create potential problems for us, though. So here's a scenario that could be problematic.
Bill where it creates a field in production.
And later, our, CI system snapshots production sees that new field and gets ready to put it into version control.
Meanwhile, build those not finished. He goes away and adds a second field, but the snapshot doesn't have that second field. So the first field added to source control, our CI system sees there's a new commit. And because it considers, source control to be the source of truth, it can deploy the entire state of source control into, our production org. What that does is clobber that second field and annoy Bilbo, and everyone thinks that CI is terrible because they can't make changes.
And if you're gonna live in a world where, changes are allowed in production and we're gonna consider production to be the source of truth, we're gonna need an answer. And there is an answer. And, Gearset CI has a, feature called Delta CI, which will solve this for you. And I've seen a, similar approach with people's, GitHub actions as well where they try to have this delta CI approach. So everything is the same up to the point where, that commit hit source control.
But this time, the CI system is gonna deploy not the entirety of source control, but just the contents of that last commit. So in this case, it takes that first field and it just puts it back into production, which is effectively no change. F one and f two are there. Sometime later, there'll be another snapshot, and field two ends up in source control as well.
So this is why Delta CI exists. It's a good concept to know about. If you're using Gear Set, tick that box. If you're using another tool, then look for, a feature like that to make sure that you can handle this scenario if you're gonna treat production as the source of truth.
Next, trunk based development and the absolutely unmissable visual pun.
So how does trunk based development work? We have a main branch in source control. We have a production Salesforce. And when we wanna work on a new feature in classic trunk based development, you, commit directly to that main branch.
But the alternative is to have a short lived, by which I mean less than a day, feature branch that you do your work on. So you create that feature branch. The dev works away, with their own separate sandbox committing to the feature branch as they do their work. And when they're happy, they create a pull request against the main branch.
The CI is able to kick in and validate, the pull request against the production org.
And as long as that's all okay, we can do the merge into the main branch, which causes the CI to actually deploy stuff into production.
And this is a really nice way to work. It's nice and simple. It's not too much to get your head around, sort of look at the pluses and minuses. So it's simple. The features are small, so we could choose which ones we, promote at any particular time. And the CI tool requirements are quite simple, which is great.
There are downsides. If two different, developers or developer teams need to collaborate on unfinished code. They can only really do that by getting it into that main branch, so into production. So it needs to be hidden for the user at that point.
There's no real data for doing, UAT testing outside of production with this kind of setup.
Sometimes the user doesn't want loads of releases all the time. They might need training on new stuff that's coming out, or we might need to use feature flags to hide things from them. So it might not always be the best approach.
So an alternative way is to have longer lived feature branches. So we're moving up in complexity as our, requirements demand it. We're not making things more complex just for the sake of it or just because we got the tools. We're actually saying, this is what we wanna do. We've got two different devs, dev one and dev two, each working in their own sandboxes, and they're gonna collaborate in source control and in Salesforce organizations by having a, integration branch stroke org and then finally putting that into production. So I've only joined joined the orgs in this thing, just because it's more complicated to have branches on here too.
So the problem with them working a way like this is that because they're long lived branches, production is gonna drift away, from the developer environments.
So then we start to have to have this kind of back propagation of changes. So what's happened in production has to move back into the feature integration org so that that's up to date, and then things need to move back to the dev orgs.
Now, this is a much more complicated approach. But if you need to have multiple teams collaborating together, then this is the kind of thing that you need to do. And this justifies the complication of using something like tier set pipelines or, DevOps center. This is why all this extra complexity is there, and people don't just do trunk based development all the time.
So, I'm gonna skip over those and, quickly. I think I mentioned most of the pluses and minuses there that we can do the UAT in a full sandbox. We can have bigger releases if that's how we wanna organize things.
We can mitigate, getting out of step by pulling changes back from production, but it does more require more advanced, tooling and procedures.
And, because we're having larger releases, there is this possibility of failure.
So summing up, there are lots of challenges, but we still do DevOps. We don't just give up, the strongest steel. It's forged in the fire of a dumpster. So it may be in a dumpster, but it's okay. And things are not so bad. Things were awful before Salesforce DX. Now they're a heck of a lot better.
And I would encourage you to really understand your goals in each situation, and that will help you to choose how much complexity you need, what the right techniques are, and how you're gonna measure your success. Use the least amount of process you can get away with and then add more when you really find out who you need it. Don't fixate on a single metric like Rob said. Take all of them together and make sure that they're all improving together.
The main one of the main reasons we wanna do this is that quality, timely feedback is what improves the software. That's why we're doing this. We wanna get stuff in users' hands and then be able to change it quickly.
So the last thing I really wanna say is that the the tools here, they can't solve all your problems. They're so much better than no tools at all, so don't go trying to reinvent the wheel with all of this stuff.
Thank you. Happy to chat to anybody on the chat here or on email or Twitter.
Please do, have a look at Nebula Consulting's blog. We've got all kinds of dev and Pardot and, general Salesforce blogs on there.
