Description
Join Matt Bevins, Principal Salesforce Engineer at Payroc, to discover how his team transitioned from frustrating deployments to Salesforce DevOps success. From the challenges they faced to the tangible benefits they achieved, you’ll see the real impact of taking a complete DevOps view.
Matt Bevins Principal Salesforce Engineer, Payroc
Emily Hollingworth Premier Customer Success Manager, Gearset
Transcript
Hi, everyone. Thanks for joining us today for this exciting session on DevOps done right, a Salesforce team journey. We're thrilled to have you here.
I'm Emily Hollingworth. I'm a premier customer success manager at Gear Sets, and I'll be your host today.
Over the next thirty minutes, we'll hear firsthand about the challenges, solutions, and transformational results that Payrock achieved with a complete DevOps life cycle.
After a fifteen to twenty minute q and a interview with our guest speaker, we'll open the floor up to your questions.
As we go through today's session, feel free to ask any questions in the q and a, portion of the chat.
So joining me today is our guest speaker, Matt Bevins, who is a principal Salesforce engineer at Payrock.
Matt brings a wealth of knowledge and hands on experience in leading Salesforce DevOps transformations.
Today's session is all about practical, actionable insights.
You'll hear, Matt and his you'll hear how Matt and his team transition from frustrating deployments to a streamlined collaborative process.
So before we dive into the details, welcome, Matt. And can you give us a quick overview of Payrock, please?
Cool. So thanks for having me. So, yeah, let's give a quick, brief on Payrock. So Payrock is a financial, technology company that provides payment solutions to businesses. We offer services such as credit card processing, point to point of, set point to point sales systems, ecommerce solutions, and various payment related technologies for merchants, agents, and partners, where our head office is based out in Chicago in the in the United States, but we have several offices across America and the rest of the world.
And we currently process around, a hundred and fifteen billion dollars in transactions last year with from around a hundred about a hundred and ninety merchants in total at the moment.
So a little bit more about my team. So I've been at Payrock, about three and a half years now working in the enterprise systems team. We're part of the technology division.
Most of most of the technology division work remotely, but, we and we have people over in, in mainly in the UK, but over in, from Europe, Canada, and the rest of United States as well.
The enterprise systems team is a small team where, everything is kept in in house for supporting, around five hundred Salesforce users in a number of orgs. We mainly obviously pick up Salesforce, but, we also are responsible for our sign in tool that we use at PayRock called Agreement Express.
And as a team, we work agile, and we use user stories, and we try to, use try to do a high velocity model where we, deploy small and often changes.
Thanks, Matt. So let's get into the heart of today's session and hear about your journey.
So, Matt, can you please describe the state of your Salesforce release process before you adopted DevOps practices, and were there any pain points with that process?
So yeah. So for us as a small team, we've got obviously a number of, different levels of people, working in the team. So we have, I think at the time, we had a, a student, placement from university who knew very little Salesforce, so we had to bring him on that journey. But we had a couple of developers, couple of senior developers, some principal, or some, business, analysts.
And so what I'm trying to get at is everyone's at different levels.
Various, sandboxes, where which were shared, and we would, I suppose, when we were doing development, we would start overlapping, and cross, meaning we'll have to either deploy someone else's change, get add a a different add change out, or, we'd potentially spin up a new sandbox if we knew there was a big overlap. And then we at some point, we'd then have to try and merge them together.
Deployments methods, I suppose, is, it was from a a with down to personal preference of the engineer, so no formal process.
It could have been change sets, could have been SFDX or other free tools available out there.
Didn't have any version control.
Lots, I suppose, you name it. We tried it. Lots of validation errors.
And, sometimes I we had occasions where, deployments that one of the team would do would potentially remove someone else's change. So, like, a field layout that was pushed through, added to a layout could have also just disappeared, and then you would get start getting questions from the business of what where did that go.
Testing would be done in the develop in the developer sandbox. Nothing formal.
No source majority of it wasn't in a separate sandbox.
We would meet our deadlines that we set ourselves, but I suppose, some deployment or the more complex ones would take, a little bit more time. So, potentially, we'd, miss, breaks, lunches, and stuff, or start even earlier to try and get things out. So stress levels could probably were quite high at times, when trying to get stuff deployed.
Great. Thanks for for sharing.
So, at first glance, moving to a more complete DevOps process, so version control, CICD, automation, it might seem more complex than having no formal process at all.
What prompted you to take on this transformation?
So I suppose we all knew that, there must be a better method, for us to get changes out there, and to deliver more value to the business by having a more precise process, having, I suppose, the ability, to be, yeah, definitely the more efficient, reliable, scalable processes I think we wanted.
And we needed to make sure that whatever we did worked for all, and I suppose that everybody was part of that journey of trying to develop this new process as, every it's not just, get on and do it. It's, we need to make sure everyone understood, bought into it as well.
Great. So once you decided to adopt DevOps, what were the steps that you took to implement this complete life cycle approach?
So for us, luckily, in technology, okay, we have these twice yearly events.
At the time, it was hackathons, but now it's off the grid.
It's what it's a events that we have, we call it, so I suppose.
So it's basically where we take a few days out from normal BAU work that we, would be norm would be assigned and then get to work on something that, isn't, I suppose, could be a, say, a process failing or, you know, look at something out of the possible, do some proof of concepts or something like that that something that we or that would deliver value to the business. So step back three years. The enterprise systems team, obviously, we're already looking at this, the DevOps and how we can deploy better and faster.
And so we, took up the we set our goal for these, I guess, three days, for this off, hackathon it was then, to pick up, to see if we could work out a better way to deploy.
So as part of this, we set, made sure we, set some goals to that we could hopefully achieve.
One process for all, as I mentioned before, we wanted to make sure it was nice and, simple or repeatable.
Version controls are gonna be a must. All our user stories are in Jira, so we, the integration's Jira. So not only our team could see where, a story was at, but also the visibility to the business. So if they knew that they've raised this, ticket into us, which turned into a story that they could yeah.
Anyone who goes to view that ticket can see, oh, it's in development. It's in this sprint. It's in it, and that sprint's gonna end on this day. So, that greater visibility, I suppose, is the was one one of the main goals.
We wanted developers to have their own sandboxes.
No more shared sandboxes where changes could be overwritten. We wanted conflict section. So if someone did deploy something that, it would be flagged, and, hopefully, we can then, take steps to mitigate any problems with that.
And once everything in sync, so no excuses that, that somebody's sandbox wasn't up to date or an environment wasn't the same similar or, you know, the same as, production.
So I suppose as part of the hackathon, we I'm I'd heard good things and had worked with Gearset previously at my old job, and they were just bringing out pipelines.
So we decided, okay. We've got three days. Let's see what we can achieve, and engaged, with the, the sales enablement team and, to basically, try and bring some of this to life.
Not gonna be not gonna lie. It was a a tool order, to try and completely rewrite an end to end process, train the team, bring it iron out any of the problems we had. Definitely, we probably took on more than we, should have done.
But, and I suppose, we as part of this, we knew that it it was a better way of working, but maybe, I suppose, Gearset, is you know, was Gearset the right tool? So, I after the we showed some promise and we did show we actually, actually, I think we won one of our little awards that we do such you know, as potentially has the most value.
We continued to work on it. We signed up a little project on our side, and said, as a team, we've we'd use the same goals we defined as this, to then have, a DevOps bake off. And that's where we went, okay.
If we were gonna pit, various tools against each other, these you know, we wanna do a common set of, deployments, between each tool and work out which is best for us. And that's work out which is best for us. And that's not only just did it deploy the fastest, was it, did we did it capture anything commit we missed, or, and making sure it was, gave us the ability to scale as a team.
So so, yeah, it's, so we continued to work on this. So every team member got involved again, and basically would report back on we basically picked three vendors.
Well, pretty pretty well known ones, and we, every team member would report back of how easy it was for them to use it, the time it took to deploy.
Did it pick up any the conflicts that we were expecting so we made sure we were doing deployments that would cause conflicts?
Did it, have room for our team to grow, and mature as a team? So, obviously, DevOps is that continuous, cycle that was referred to in the earlier keynote.
So it's we knew, and it's not something you just stand still with. We we know that as you learn more, you, mature as a team that we would want to adapt and achieve more, the end goal. So after let's say. Right?
Long story short, we pitted each against each other.
We revisited Gearset and took, some steps, took our learnings from the first time and then made sure we, I suppose, gave it a fair, comparison against the others as well. And long story short, GearStack, was chosen by the team as it gave us, all everything we needed and, gave us the ability to grow as a team. So, yeah, it was really good.
I think yeah, it Yeah. And everyone it allowed everyone to play a part in what's, a part in to achieve, our goals, I suppose, and and to achieve more from that.
Awesome. Great. And, I love how you all had your goals and were very thorough with your bake off and making sure that, you know, the solution you went for really worked, like, for the whole team and, like, everyone involved. That's really great to hear.
So once you'd adopted these DevOps processes, what tangible benefits did you see for your team and your processes?
So, obviously, one of our goals was the speed to deploy. And, so fast deployments, not only in time it took to remote, but to make sure that everything was kept in sync, in the pipeline.
We obviously when I spoke about before, we only, you know, only having that dev dev environment and potentially just reporting from there straight to production.
We now have, an integration at org, where where we do our initial checks. The bit then we have a after we've done that, it gets promoted to our u UAT, user acceptance testing. So that's when the business comes in, or we do some we got some more thorough data in there, to, you know, give more real world examples.
And we also have a hotfix environment. So that's a full copy sandbox where we can, if we experience problems, we can bring that in and, and, you know, get a change out as quick as possible, to, you know, fix forward. So it's obviously important to keep have all of them in sync, but also the dev sandbox as well. So definitely, that was the, I suppose, the next bit is making sure that everything that we have is able, to keep in, was kept in sync, and that everybody's sandbox is, you know, they had they knew when they needed to back promote something. So when a change hits production, that's when we know that that it gets, back promote or a port sort of a an update gets created and goes goes for the developers to then, back in, back promote into their sandbox.
And, as part of that, it will check to see if anything they're working at the moment has any conflicts and if it you know? So, obviously, we will know be able to should be able to catch any of the the work being overwritten or, any conflict you know, different conflicts that may have been missed.
Tests, testing, in our environments, obviously, if we've tried to shift left, so detect problems earlier.
So with the the structure of our pipelines, you we've now shifted quite a bit more left rather than letting changes get into production.
We've, we, yeah, we make sure we test quicker and faster.
I'm sure I can go on, but I suppose they're the main benefits.
Great. Some really awesome, benefits that have come for the team then since since adopting this.
What was the biggest cultural or operational change that you noticed as a result of this transformation?
I suppose it's the reputation we built.
As obviously, we everyone knows Salesforce is quick to potentially, ship changes, but we not only, I suppose, get, got got the rep I suppose we built that reputation that if the business needs to, act quickly on something, for a process change or something like that or, something that something something they need to capture. We were able to turn around, a business need into a story, into a, into a build, and get it through the pipeline very quickly. And so then short no one likes the short deadlines, but we've, proven that through the DevOps model that we've, adopted that we could adapt to business needs and ship very quickly when needed and, and, yeah, deliver that value the business needs as quickly as possible.
That's wonderful. That's really great to hear.
Well, thanks for for sharing all of that, Matt. I think before we move on to questions from the audience, in in a couple of points, what advice would you give to a team that's considering this move from no formal process to this complete DevOps, life cycle?
Yeah.
I think the main one is do it. Get on with it.
It might seem daunting, but the end results that you get will, is worth any pain to treat yeah. To I suppose we learned to we should have started small that, you know, we shouldn't have just gone straight into that, hackathon and tried to change the whole end to end deployment process as well, in in one go.
Making sure you know, you can do little bits. You don't need to do full blown automation testing. You can, or put every bit of metadata, that exists in the Salesforce org into source control. Think about the things that, you want to have in source control and the things that don't aren't likely to change in, production or things, you know, dashboards, reports, and, list views. If your users have got access to create them, having them in a in a, source control where they're not automatically, you know, kept in mind, doesn't really add too much value.
I suppose try different solutions. There isn't I would there's there isn't one solution that, is best for everyone.
Find the one that's right for your team. I will say that free versus paid, doesn't always, I suppose the free version doesn't always have that cost savings because the time and effort it takes to try and maintain the support for a free tool, and the pain of when it doesn't work is, sometimes not doesn't outweigh the cost of aim for a solution.
And it makes sure whatever solution you go for, there's room for you for the team to grow and adapt, to, you know, different requirements that could could come, as you mature as a team.
Obviously, adoption, of the tool is, crucial.
So bring not only your team, but the company on, the business with, on that journey. Acknowledge that velocity could be slower at the start because you're having to take a step back. You need to bring every you know, upskill everybody on this new tool, but the quality, the speed will, will come as, as people as the team gets, used to it, and it becomes business as usual.
And finally, training, depending on the levels of within your team. There's plenty of training out there. So, if you, obviously, Gearset's, DevOps Launchpad is a great way to learn all about this. But there's also stuff on Trailhead about, GitHub, for example, and how what a pull request is. So it's good to to get that level ground so everyone has that, knowledge. So, yeah, definitely, think, bring you or do all of that, and, hopefully, you'll be successful in there.
Fantastic. That's some great advice. Thank you for sharing that.
So we have, a few minutes left, and we have some questions from the audience. So, the first question is, who on the team is responsible for keeping your sandboxes in sync? So individual integration, UAT.
Yeah. So, the answer is it's down to the developer.
They know what's in their work, what they're working.
As, obviously, as we go through the pipeline, that is what's progressing to production. So as soon as it's merged into production, it's then down to the developer to, the individual developers to get a change, to check make sure their sandboxes are in sync.
If any conflicts or anything like that, occur, we're quite a small team, so we're quite respond we speak you know, could put a a message out on, Microsoft Teams in our instant, instance to say, you know, how do I deal with this? Or we and we've we've started doing pre and post deployment steps. So if there are any things that someone needs to be aware of, that's, you know, KSS allowing us to, put that in as part of the the change, but, you know, it's being deployed. So, yeah, it's the whole team is responsible, but it's made the developer environment is down to the the developer themselves.
Great. And so, a follow-up question from someone else, is along those lines, how often are you refreshing sandboxes?
So this might be a surprise, but we, if I think of our current pipeline, last time our developer sandbox still got refreshed was probably coming up a year ago now. I think it was March twenty twenty three when we last refreshed them all at the same time.
But the process we have in place means technically we can refresh them at any point because, GearSet allows us to, you know, takes that, dependency out.
So, yes, we can refresh, but we haven't found the need to.
Thank you.
And a follow-up to the follow-up is, wow.
You haven't refreshed with Sandbox in a while. Do things go out of sync?
No.
We haven't found we haven't had any problems with that promoting.
Obviously, we know that, Salesforce themselves do releases a couple of times a year with the summer, spring, and winter releases. So we know that, we have to, try and, at some point, we'd, and if we enable standard features in production, we'll have to refresh the sandboxes.
So, but we haven't seen any main problems with that.
Brilliant. Great. We've got two more questions.
So penultimate question is when making a decision to roll back a bugged feature or fix it in a hotfix or lower orgs, What determining factors go into making that call?
So for us, depending hopefully, we caught that bug before it got into production, but if it has got got, gotten into production, it, it would be the severity, of that instant, should we say.
What impact is it having? Is it something we can quickly just, if it's like a new flow that got deployed, can we just switch it off into production then fix forward?
Or if it's, if it's something that is having a major impact, it's I don't know. I'm trying to think of a worst case example. Emails not coming coming in on a case or something like that, then we would you know, the the factor of create you know, causing that outage, we would, then roll back and then look to fix forward, after that, or to revisit the problem, should I say. So it really is, I got it comes down to what what is the root cause of it and how quickly can we fix forward and, and such word, we've rarely have we normally catch try catch them the most issues in a lower environment, which means we don't cause that impact. So we don't have to, roll back that often Yeah. Or or or look to fix forward, should we say.
Makes a lot of sense.
Last question for today's session. How did the implementation of CICD change your sprint planning process?
So I suppose when we originally picked this up, the it I suppose we were as part of DevOps, we were also looking at how we implement, we implement sprints and our way of working as a whole. So it sort of came hand in hand. So we and as a team, we adopted sprints at the same time as rolling out this. So it was a big change as it, and we had to bring once again, bring the business on, but, we'd, you know, point point the stories that we do. So we don't, you know, just have one huge ticket that takes forever to, to do. We try and break them up into small chunks. And, you know, I refer to that high velocity model is we, we actually now deploy more, I would say, three or four times a day into production.
These are small little changes. So when a when something is in sprint and it's complete and it's being tested and the business has potentially signed off it, We don't hold it back from, until the end of the sprint. We, you know, deploy it as soon as we can, and that means we are deploying three or four times a day to production. Some of them changes might be little things that are gonna build up to a bigger change that people don't see until it's ready to that, you know, the big launch that, and, you know, different bits like that.
But that whole process of implementing, how do we change our sprint planning, it was part of that journey of us adopting DevOps. We were like, how can, you know, how can we be better as a team, you know, to works in smaller chunks, to be able to buoy more often, keep things in sync, so we don't have then conflicts further down and you know? So it it is that whole process, and for us implementing a more agile and more, that agile process of get getting things out quicker in smaller chunks and being sort of responsive to business needs and things like that, it's meant that the sprints themselves, we don't we don't necessarily set. We must achieve x amount of points or something like that. We achieve what we've got in the pipeline, what's ready to work on. And, if it's starting to start it, but we you know, the sprint's ending, it just rolls over to the next one.
Brilliant.
Great. Well, thank you everyone for asking all your questions.
Hopefully, that's been, illuminating. And thank you so much, Matt, for sharing your DevOps story with everyone. It's been a great session and, yeah, loads of really interesting advice and tidbits. So, yeah, really appreciate your time, and thanks everyone for joining.
