Description
In this session recording from the Gearset DevOps Summit 2022, Dan Appleman (CTO and Co-Founder, Full Circle Insights), breaks down the challenges of building a DevOps culture in the Salesforce ecosystem.
Dan covers:
- How both developers and admins modify metadata — for DevOps to work, everyone has to play!
- The dangers of focusing too much on tools — tools support culture, they do not create culture
- And top tips for getting started with building a DevOps culture.
Learn more:
Transcript
There we go. There we go. Hey, Dan. How are you doing? Oh, doing well. How are you?
I'm good. I'm good. I'm excited to hear your session. So with that, I'll, let you take over.
Alright. Let me, quickly share my screen.
And, welcome to the challenge of building a DevOps culture in the Salesforce ecosystem.
You know, I was a little little bit worried because I didn't know what people would say before, and I was sort of afraid. What What if they cover all the things I'm gonna say? But I think that, I think you're gonna find this interesting and and hopefully some fun as well.
For those who don't know, my name is Dan Appleman. I am the CTO and cofounder of Full Circle Insights, a company that specializes in marketing marketing analytics, where I also do development, participate in DevOps as well. I'm a Salesforce MVP, and I'm the author of a book called advanced Apex programming, and also, create courses for Pluralsight on the Pluralsight platform, including courses on Salesforce technologies.
So the genesis of me being here today comes from an invitation I received to attend the, the gear set summit that's later this month. I did decline because given the state of the world and so on, I just didn't feel comfortable committing to attending and speaking at a conference. But one of the things that fascinated me was that the day I would be speaking is April Fool's Day. And I thought to myself, woah.
If ever there was a subject that lends itself to April Fool's pranks, it's DevOps. So I'm gonna share with you right now some great April fool's pranks that you, as somebody who cares about DevOps, can pull on your teams, especially your admins. Now I I do wanna say that anything you do is your responsibility. Use these at your own risk.
I I will I you know, it's not it's not my problem if bad things happen, but, here's some things you can do. So for example, they take the form of, like, a memo or an announcement. So you can announce to, to everyone, especially the admin, say, from in following best DevOps practices, no metadata changes may be made directly in production other than maybe personal reports, and that all changes must be built on a development sandbox, be reviewed by a team lead, and then deploy to a QA sandbox and then reviewed by IT before being pushed to production. They are gonna scream.
They're gonna go nuts. I mean, could you imagine the reaction they're gonna have? What? I have to do all this stuff just to build a workflow?
So this is a great initial pipe, but you can raise the stakes. For example, you can increase and say, you have to now go through all of these steps. You have to build on the development sandbox or scratch org and just call saying the word scratch org is gonna freak out a bunch of admins. You're gonna say, never heard of that. It all has to be committed to a source repository.
Right? Just think about it. You're an admin, and you can't deploy a workflow until it goes to a repository. You don't even know what a repository is.
They're gonna love that. It all has to go through a code or metadata review. So somebody actually has to look at what you've done. You have to run every single unit test in the org, not counting manage packages because we don't care about those.
But your own unit tests just think about how an admin react would react. Say, wait a minute. I'm gonna change a field to a restricted field, and you want me to run all of the orgs unit tests for such a small change? Right?
They're gonna hate it. They're gonna totally freak out. And that code has to be deployed from a repository to a QA sandbox and then to production. And, again, repository.
I have to tell you, this is going to this is going to absolutely drive them bonkers. It's gonna be the greatest prank watching people react to this kind of stuff. And it's gonna put them in a you know, know, they're gonna be stressed because it's gonna put them in an impossible spot because, you know, a manager comes and say, here, I need this quick change to a business process. Just tweak this workflow or something or or a pick list values or so on, and they're gonna say, well, I'm sorry.
We have to go through this deployment process. And, I mean, it's gonna be awesome. Now if you wanna take it to the next level, right, right, take all of those things you saw earlier and say, okay. They this process has to be followed by all of your admins, all your your developers, every single department and division, and any consultants and contractors.
Right?
I mean, if this does not lead to the great resignation in this org, nothing will. So these these are fantastic, fantastic DevOps pranks that you can play on your Salesforce teams. But, again, I stress, I'm not responsible for any of the consequences if bad things happen.
Now it is possible. It is just barely possible that some orgs would react to these differently. They may find they're not funny. And and not that they're not funny because there are pranks and some people have no sense of humor and some people just don't like pranks. Not funny because they don't understand the joke.
They might say, well, hold on a minute. We do all these things. I I don't get it.
Or they might say, well, you know, this is a small change of process from what we're already doing. Again, it's like, what what's the big deal? Or, you know, they may not even see that it's a prank. Right? Because they'll see these things and they'll say, well, I can't imagine any other way of doing these things. This is just of course, this is how you deploy in the Salesforce ecosystem.
If you get that reaction, you have a DevOps culture because that's what culture is. Culture is, when you do the right thing and you don't even have to think about it. It's not even a question. This is how we do things.
That's an expression of culture.
So where where is Salesforce DevOps today? I don't have a survey, but this is this is my sense of what it is in in working with, our many customers and speaking with many developers and so on, and, again, from a cultural perspective, from a coast a perspective of DevOps culture. So the first thing to recognize is that for DevOps to work, everyone has to play. And you've heard this earlier today where somebody commented that even if even one person doesn't follow the process, that's enough to jeopardize all of the benefits you're getting from DevOps, which is absolutely true. But in the Salesforce world, it applies to both developers and admins. And that's because what we care about is not code. We care about metadata.
And if you really wanna have DevOps in the Salesforce world, you must be talking about metadata and not code. And the reason we know this is true is because it is possible for an admin for making a metadata change in the UI to break your process, to break the code process just as it's possible for somebody in code to break something that's been done in automation.
Because they can break each other, because they interact so tightly, you have to talk about both. And I I really wonder when I saw the survey earlier this morning whether that discussion was in place. When people responded to the survey and said, yeah. We have DevOps, were they thinking just about Apex, or were they thinking about all of their metadata?
What metadata counts as DevOps? And in my example, I talked about excluding, you know, personal reports. There might be other things you can exclude. But if it has the potential of breaking code, you gotta include it as metadata. And, of course, Apex code is metadata. And I'd love to see in a future survey, some discussion, some some way of addressing metadata and making sure that people are responding or asking, people who are responding to the survey, do you consider all metadata? If you considered all metadata in DevOps, do you still think you're doing DevOps?
Interesting question.
I suspect things will be a lot worse than the survey showed. So we're gonna split it up, and we're gonna talk about the state of DevOps culture both from a developers and an admin perspective.
Now when we talk about the developers, honestly, I think it's mostly good news. First of all, we have SFDX now. We have the Salesforce development experience, which means it's scriptable. It means we have the ability to track UI metadata changes and pull them into repositories.
So it is now possible to do DevOps in Salesforce.
The other thing that's good is that we are gradually seeing an increase in developers coming into the Salesforce world from outside of the ecosystem.
And the reason this is important is because outside of the Salesforce ecosystem, by and large, developers have DevOps culture. Right? If you were to take those prank questions and go to a average random development team in, at Google or Facebook or any other major company and said, you know, here are the steps for deployment, they'd sort of nod and say, yeah. Alright. That sounds pretty much like what we're doing. In the software development world, having a culture of DevOps is pretty normal.
That has not historically been the case. Salesforce is catching up. But as these external developers come into the ecosystem, they're bringing the culture in with them, which is a wonderful thing.
It's not as good in the Salesforce world as in the outside world because we do have a lot of developers who grew up within our ecosystem being taught how to build things into developer console and use change sets and so on and then learning Apex. So I I it's not as good, but it's still pretty good. And it's good enough so that I would argue that at this point in time, if you are creating Apex and building code as a Salesforce developer, if you are not using SFDX, if you are not using a repository, the very basic DevOps process don't talk about CICD. But if you're not doing those things, at least, you're not a Salesforce developer.
You don't count. We're gonna strip away your certificates. You don't deserve them. So, that might be an extreme statement, but I believe at this point in time, that is a fair requirement.
That is a fair request. If you are a Salesforce developer writing Apex code, you've gotta be using an IDE. You've gotta be using SFDX, and you've gotta be committing to a repo. Basic ground rules.
Going to the admin world, things are not so cheerful.
If you were to go and say terms like IDE, scratch org, CLI, CICD, repository, I think the reaction in many cases will be, I don't even know what those words mean. Are you kidding? Right?
They are in pain.
Right? Our system admins are in pain. They know that they don't know why people made changes. They can't keep track of that.
They know that, they don't know, you know, what metadata, what workflows formulas are actually still being used and what they're doing. They know that people are overriding each other's works. They know that they can work in a sandbox and have it overwritten during a refresh. They know the pain, but they don't know that there is an answer for this.
Right? They don't even know the options.
They are so they are they are so early in the learning curve that in many cases, they don't even know that the learning exists. They don't even know the journey is possible. This is how bad things are in much of the admin world right now.
And I know this because I speak to admins, and, they don't know these things. Right? So we have a lot of work to do. And, again, we need the admins. We need them. They have to be part of DevOps because for DevOps work in Salesforce, everyone has to play.
They are still teaching clicks not code without the obligatory pulled the metadata into a repository.
Right? It's not even on the radar. It's not part of the discussion. And testing on automation is primitive.
Right? It's mostly manual, and there's almost no testing at scale. And I know this from personal experience because it's almost on a weekly basis. We will have a customer who will reach out to us and say, well, we just tried to upload two hundred records, and it failed, and it must be your fault because, you know, the first thing you do is blame a managed package.
That's just the way the world works. We will work with them. We will look at debug logs. We will do, an analysis of it.
And about you know, sometimes it's it's changes in process. Sometimes our tool is involved in it, and we can show them how to to to deal with that. But nowadays, about seventy percent of the time, maybe eighty percent of the time, it's process builder.
They built a whole bunch of processes, and they never tried to do anything at scale, and it blew up because process builder is evil. If you don't know that, now you know it. Switch to flow. Do anything, but don't do process builder.
So this is this is a problem. This is this is a problem with especially on the admin side. So how do we deal with it? Now as technologists, we have a tendency to focus on tools, and we get very excited about tools, and it's very tempting because you like to think, wow.
I can go out and buy this tool, and it will solve my problem.
And tools are important, and tools do solve problems, but tools do not create culture.
Tools do not educate people. In fact, tools create a a need for additional education because you need to understand the concepts and the reasons for the tool and then how to use the tool itself. So it is important for tools to be part of the picture, but they are not the starting point. They are not what you need to use to create culture. Once you get the tools, they will help the culture evolve, but they do not create culture on their own. You can't buy culture.
So how do we create a DevOps culture, especially in a world where we need to address creating a DevOps culture among system administrators in the Salesforce world?
Well, to start out, let's talk about, the challenges of creating a culture. Why is it difficult? What are some of the characteristics of human nature, that impact this and make it such a challenge? Well, first of all, as human beings, we are very bad at giving up a short term benefit in return for long term gains.
If we were good at this, there would not be a seventy two billion dollar weight loss market because, you know, what is the challenge of weight loss if not giving up that, you know, chocolate cake right now in return for living an extra year? Right? I don't know about you, but there are days where I'm gonna eat the cake. Right?
So we're not good at that. Also, we do not like being beginners.
We like to feel competent. We like to feel comfortable in what we're doing. And as soon as you say, oh, here is a new process. Here is this new thing called DevOps.
You're a beginner again. Right? You're starting. I know how to do things. I'm comfortable with the way things are.
And now you want me to learn something new, do something different?
That's not easy. That's not something we'd like to do. We don't like change.
We tend to prefer easy over hard. Right?
All of these additional processes to deploy make things harder. It makes it harder for me to get that change into the org, Sometimes a lot harder. It's gonna take more time.
And we don't like conflict and driving change. Creating a DevOps change inevitably involves at least some level of conflicts. You know? I am trying to persuade you to do something different.
You know? It can be very friendly con conflict, very productive conflict, but it is conflict of a in a sense. And sometimes, of course, it can be more significant.
Now this applies to companies. It's not just individuals. Right? Companies, we need to deliver profits. We need to deliver work. We need to get things done.
Creating DevOps doesn't do that. Right?
DevOps the benefits of DevOps are return on investment.
Part of that investment is money, but part of that investment is time to build the DevOps, to establish the processes, to learn all those things. And the time you spent learning all those things is time you're not spent delivering things today, you know, the work the company needs to do. Beginners cost money. Right? You have to train them, and they're not as productive. So all those people who are now learning the new process, they're expensive.
So you're not just dealing with human nature. You're dealing with corporate nature.
And DevOps has costs. Right? You have to pay for tools. You have to pay for the infrastructure on which the tools are running or or the cloud services.
You have to pay to learn how to use them, the the time that it takes to learn. The processes themselves, DevOps has costs. It slows things down. It it does.
You know? Compared to making a simple automation change in production, going through DevOps, even with automation, is going to slow things down.
There's the cost to establish it. The cost of training I mentioned, it needs to be managed. Somebody has to be paying attention, and it has to be maintained.
You don't just create DevOps and walk away. Somebody's configuring those tools, watching them. You know? We have a Jenkins server doing automated testing, and sometimes things get weird, you know, with SFDX.
And somebody has to go in and, you know, reset things and make some changes and clear out some caches and and get things up and running again. So, you know, the processes have costs. And all of these costs, everything we're talking about goes against human nature. And because it goes against the human nature, because it goes across those short term corporate interests, it is hard.
So how do you deal with that? Well, I wish I could say I have a magic solution that there's a silver bullet that's gonna deal with this. But I'm gonna point you towards some some concepts, some directions, that I think will help. First of all, think about the carrots.
Right?
Talk about the benefits. Right? That's the starting point.
Remind people, oh, yeah. You have this pain. You're trying to figure out where the where that metadata came from or who wrote it, why did it happen. You know, if we have DevOps, you would know that.
You know, you wanna restore the the the state of, of of a change or find out what that workflow looked out before that before you made that change. You know, if we have DevOps, we could do that. So you start talking about it, plant the seeds.
You're gonna have to get management buy in, because you need management to participate in it.
If if if people are penalized because there was a delay due to DevOps, right, you're not gonna get DevOps. So, it is really important to get the managers involved, and you need to create incentives. People need to be rewarded for, for following DevOps processes, and that means creating metrics around it. And, you know, it should be part of employee reviews. If if you're a system admin and you're reviewed, one of the questions is, have you been following the DevOps processes? Have you helped build the DevOps processes?
On the stick side, you need policies, right, that are expected to be followed. So you need policies around DevOps. And you can start with the simplest things. Right? Don't, you know, build a whole DevOps solution and then try to do it, you know, deploy. No. It's like start with the policy around everything has to be in a repository.
For example, everything has to be every metadata change has to go through a sandbox. No more changes in production. Right?
You can enforce policies policies through org security, for example. Not everyone needs to have the ability to author Apex. Right? Give the give people that permission only on the sandboxes and so on.
Enforce review prod policies, right, so that you cannot get in trouble due to delays and but you will also get in trouble if you bypass DevOps.
And build it into third party contracts.
Right? Make sure any consultants you hire, any consulting firms that they are signed off, that they have to follow your process. You know? And they have to write real unit tests. You know, there's a legend of somebody trying to get that code coverage you need to deploy Apex by creating hundreds of lines of I plus plus, which does nothing, but it increases your line count so it's easier to get that percentage. And I thought it was a myth until one day, I went into a customer work to help them resolve an issue, and I found thousands of lines of I plus plus in the class, and I just died. I mean, I I can't even describe my reaction to that.
It was it was I was I felt very sorry for that company. So enforce that. And to the degree that's possible, create automation tests as well because manual testing is very time consuming.
Now let's talk about building culture in general.
Lots of things you can do. People talk about building culture with with with books, with papers, and things like that. But at the heart of it, there are two things you need to do to build culture. One is you need to talk about it. You need to write about it. You need it to be part of every conversation as much as possible, and you need to set the example.
If you have managers or executives who can bypass the process, you don't have process.
And some people will react and say, well, we need we we can't create all of this DevOps process because we need the ability to move quickly. We need to we need the ability to push emergency fixes.
That's okay because any good dev op DevOps process has an emergency process as well, which accelerates things in the case of something that's urgent. So you create processes around those needs as well.
If you're communicating culture and if people are role modeling it and setting the example and following it, then you're going to get culture. Right? Start small and build on that. So let's talk about getting started. Somebody asked previously, what is the very first step you can do in terms of building process? And I'm gonna start with two things.
And, they may seem odd, right, because the natural response would be, you know, repository or or, you know, automated back ends or some technology, but it's not technology.
I believe that to build a foundation of of any culture, of any technology, of any practice, it's important to know the foundations. So what are the foundations? Where do we start? Well, the very first place we start is through learning.
So I actually did a course that was my my small attempt to try to, help with this. Right? So I I created this course on Floss, I called Salesforce development getting started. And what it basically says is, here's how you get started, you know, develop org and so on.
And here's what metadata is. We're gonna talk about metadata first. And we're going to talk so because if if you get your admins thinking about metadata instead of workflows, processes, and and automation and database schemas. If they're thinking about metadata, then it's not too hard to talk about backing up metadata.
Right?
And the other thing this does is it starts its SFDX first. So starts with Visual Studio Code and the concepts of pulling and pushing not only code but metadata. Right? So the there are examples in there of, okay.
You just created a workflow in a UI. Here's how you pull it into your metadata store. And it's really designed it's it's called Salesforce development getting started, so it's designed for beginning Salesforce developers, but it's also designed for admins to learn SFDX, to learn about metadata and the idea that all of that automation that they're building with UIs is actually metadata. So, you know, that concept, whether it's on Trailhead or my course or any other source that helps admins understand that, hey.
We're all building metadata, and there's this thing called DevOps starting with source control that can help you get a handle on the pain that you're feeling, that you're experiencing, that's a foundational thing. I think you are gonna have to get that basic core concepts spread throughout the Salesforce teams, admins, and developers in order to even begin with any sort of, of DevOps and building a DevOps culture. Second, I think focus on governance. Right? Whatever you start to do, whether it's just a repository or skipping change sets or or having a review process, whatever it is, build the governance around it so that everybody is paying attention to it. Right? Because if you have the governance in a good place for a single process, you can build on that.
Right?
And if you build on that and you you build the pros you build the process and you build the governance around it, you're going to start getting culture around it as well. So I just wanna conclude with a little bit of a discussion of the difference between process and culture.
So if you have process, which is a a good starting place if if you don't have either, you might say, okay. We now have government. We have process. We know the policies and procedures that we need to follow. We're gonna know we're gonna be rewarded if we follow them. We're gonna get in trouble if we don't, and we know that everybody's going to follow the process.
And that's good.
Where you get to culture is you're taking to the next step. You say we have governance in the process. We support and we respect that process.
Right? We understand the reasons behind that process. We understand the goals, and our decisions support those goals. And our decisions is the keyword here because what it means is that when facing a question that is not covered by the process, people will make the right decisions because they understand the reasons. They understand the goals. And in making those right decisions, they will evolve the process as well, and they will be happy when new processes come into play and will adapt to them.
