Description
In this session at DevOps Dreamin’, Jami Medina, DevOps Manager at Upwork, and Nathan Galteland, Director of Sales Operations at Upwork, share how they have grown and developed the DevOps program at Upwork using a globally distributed workforce. They discuss how they use tools like Gearset to manage rapid deployment, as well as the benefits and challenges of managing a DevOps program with a hybrid workforce. They also discuss:
- Best practices
- Collaboration tools
- Processes and lessons learned from evolving the program over time
Learn more:
Transcript
Alright. Thanks everybody for coming.
Check one two. I'm good.
You good? Mhmm. Here's okay. Great.
So I'm happy to introduce us. I'm Jamie Medina. I've been working the Salesforce ecosystem about twelve years in the dev space since twenty eighteen, and I currently serve as the DevOps manager for Upwork and help lead our Salesforce team with Nathan.
Yep. My name is Nathan. I'm, director, at sales operations for Upwork.
And, yeah, I help, help manage as well.
Here to talk a little bit about how we've built our team and how we've grown it. Certainly a big part of that.
So we're gonna we want this to be a really practical discussion today about how we've solved real world challenges in an older complex org. So we're gonna cover some things today where we talk about how we set up and structured our integrated team, sourcing it for global success. We'll talk about our communication strategies and using the tools we have to help really effectively deliver, our our service processes. And then we'll take you through, a real world problem that we're solving using some of our CICD tools. We are Gear Set customers. You'll hear us talk about GearSet in terms of how we're using the tool today and and how we've used it as we've evolved over time.
Cool. Alright. Show of hands, who's heard of Upwork?
Yeah. Most of us? Okay. Cool. Upwork, if you're not familiar, it's kinda like, it's kinda like eBay for talent.
Like, you can go online, you can find a job if you have a great skill. You can go online and find somebody to do a job for you if you have a great need. So that's sort of how Upwork works at its at its core.
Us, specifically, we work with the sales team at Upwork. In addition to just kind of, you know, anybody being able to sign up and use Upwork, we also have a enterprise product where Microsoft can sign their license, for instance, and then they can give access to the Upwork platform for all their employees. So as a Microsoft employee, as a manager, something like that, if I have a job I need done, I go on Upwork and I can hire it and it's under the Microsoft account. So we have a sales team. They're trying to go and sell those deals to try to get, more people kind of, you know, on onto those type of, contracts. And that's sort of what our, Salesforce instance is mainly supporting, is that kind of motion and efforts.
So back in the day when, I started, I rolled up to a senior manager and I had a dev and I had an admin on my team. First and foremost, no one cares about the senior manager because they're not here. So let me get them to go away. Alright.
So I had an admin and I had a dev, and that's sort of how I started. Now we had probably twenty to thirty salespeople or something like that in our our Salesforce instance. Our Salesforce instance actually was created in two thousand eleven. Most people were still on classic.
Had a lot of lot of challenges. And this is and I started, about six years ago, so this would be a twenty eighteen. So a seven year old instance now, lots of data, lots of baggage. And I have one admin, one dev.
Now we are talking about a globally distributed team, so this diagram doesn't quite do it justice. In reality, I'm here in Chicago. My dev is here in Indianapolis, and my admin is over there in Thailand having Mai Tais.
Mister, Thailand, he he was a great guy, but, yeah, he tended to, you know he showed up and we were all asleep. No one really knew what he was doing. It was a little bit of a problem because he was also like our support person, So, like, his hours didn't quite line up with everybody else. So we needed to find more admins, and that's actually where something like Upwork is really strong.
We do use our own product. We're kind of customer zero. We try to use Upwork ourselves. So I would go online and I would look and go find some other admins.
Our admin actually gave us notice that made that made that decision easy, and I was trying to find some new admins. And over the course of six years, we found admins all over. So I've used admins in, Michigan. I've used ones, also in Indiana.
I also used an admin, in, in Mexico. That's actually our current admin is currently there. So, finding talent, you can see though that I started finding admins not in Asia. And that's mostly because of the time zone difference.
We'll talk a little bit about considerations that we've kind of learned as we've kind of been building out the team, but that's why you see, you know, lots lots lots more kind of, you know, North American center because that's where our sales team was. And I should point out, we don't have a global sales team. Our sales team is primarily focused in North America, so that's why this worked really well for us.
Then we need some more development help, so we've started finding developers. Now developers, that work was a little bit more asynchronous. Like, we can kinda give tasks. We can give kind of objectives.
We don't need to necessarily have that be in real time. So you'll find that I actually found, developers in India, found developers in, Uruguay, developers, I think I saw only two. I think they're more coming. So so finding some developers around the area, moved our our our our person in Indianapolis.
We moved him to a senior developer at this point. And we started building out the team, which is pretty good, getting all over the world.
At this point, I moved up to being a director. I needed somebody to help manage all this, you know, dev work. So we brought Jamie on. So we have our dev ops manager. Jamie, at this point, I think he was started in Alabama.
Is that where you work?
So found her. Again, on Upwork, went to Upwork dot com. Found, I I made a post about, like, if you bleed in iterative sprints, then you this is the job for you. And she found my article funny, and she just applied for it and hired, her as our scrum scrum master to start and then, kind of evolved into a DevOps, manager position.
She came on and said, Nathan. Said, yes. Said, you don't do any testing. Like, that's true.
So we hired a QA person. So we went to Canada, found a QA person, again, on Upwork dot com. They said, Nathan. I said, yes.
Said, we we don't know what anything is because you didn't write anything down. I'm like, well, that's true. So we didn't find a technical writer from the Philippines. So we get as we needed things, those things came up, we just kept hiring.
And again, went on upwork dot com, went and found these people. Upwork dot com, I'm not here to sell it to you, but it is great. You can see people's past history. You can see people who are well established, people who have good feedback.
So finding, you know, these technical writers, finding these developers, we could do it with a lot of confidence because of the background we were able to see on all these people. And finally, you know, time to roll out CPQ. Well, my team's like, what's a CPQ? So we went and found a CPQ specialist.
He was in India. So again, as we need things, as new stuff arises, we are really able to build out this team. So that's sort of the evolution of where our team has kinda come, and that we run a very distributed development network from all over the place. Jamie's currently living in Nashville, but again, still not all local.
I live here locally in Chicago. And this is sort of kind of where everything's been sourced from. I'm gonna pass it to Jamie, who's gonna talk a little bit about what day to day oh, wait. Actually, no.
I think I have more slides.
I do. So couple of things to think about when hiring, things that I've learned kind of through this process. Some of these will be obvious. First, we already talked about it.
Time zones. Time zones make a big difference depending on what the role is. So if we are looking for like, we have a Slack channel, which is where everybody goes and ask questions if they have it. If I want somebody to man that Slack channel, they need to be somewhere near the time zone of everybody who's gonna be asking for help.
So I can't hire somebody across the world for that one as easily. Time to respond if that's important. Real time meetings, if you need to, you know, interface with them a lot, and also direction or training requirements. It's really hard to ramp up somebody if they're on the other side of the world and teach them everything they need to know if, you can't kind of make that time work.
Does the person have prior experience working outside of your employees' time zone? That's actually a really big one for me. When I was interviewing people for these roles, even though I was maybe hiring them in India, which is like an eight hour, nine hour time difference, I found some people who basically made their career on working for American companies from India, and therefore, they started their work day at like six PM, and then they worked until two AM. That's just what they did.
That was their lifestyle. So just because they're in that area, it doesn't necessarily mean they can't make it work. They sometimes do. I have one guy, he's been with me for like four years.
That's what he does.
Next, communication.
Fluency matters.
It it just does. It just especially when you're working with somebody who's not in your time zone, if you're working with somebody somewhere else, it's really challenging if there's a language barrier there and you give the requirements, they don't quite understand them, and then you end up with something that's not what you wanted and it takes you a whole another day cycle to fix the requirements and try it again and try it again. So, I will say that, like, for me, I always look for someone who I can communicate really well. If I had, like, some confusion or something in our interview process, or I had to say something two or three times before they kind of understood what it was, those are red flags to me. Like, this might be a problem. We maybe shouldn't go forward. Another thing I'll point out is cultural differences.
It's always fun when you find out that a different country has a holiday for a week that you didn't know about, and so everyone's gonna be gone. So, you know, you you have to you have to know that stuff. You have to be aware of that. You have to talk about that. But, some things are maybe even less obvious than, like, you know, holidays and stuff, which kinda makes sense.
I found, like, just different cultures just have a different way about working. Like, some some, like, I some some some people from some regions have, almost an aversion. Not not an aversion, but like an they don't naturally communicate or, like, like, comment on things. It's funny.
It's like everybody I hire in that region, it's similar. It's just like it's just not it's just not what they do. Like here, I'm very much telling my people, like, comment every day on what you're doing. Make sure that we see progress, that we show progress.
It's very important. Over there, it's like, well, I'm making progress. Right? I logged hours. What do I need to, like, type a bunch of comments there?
So this is like, it's a cultural thing. Like, hey, you gotta learn. Hey, this is actually what it is. Some of that's internal, like our team's culture, like, they have to learn some of that.
But also, I have found, different regions have different, like, you know, takes on this.
Another thing is, this one was really interesting to me.
There are some regions where job security is a really big deal, a really big deal. So in those areas, if I'm, like, a week late in renewing their contract on Upwork because I have to go and, like, click some things, it's, like, a huge deal. Even though I can do it and they can type in all their hours for the previous week and it's not a problem, and I've already told them, hey, no issue. Hunters can be renewed.
I just haven't gotten down to it yet. It, like, legitimately gives them stress versus other people, in other regions that they're just like, yeah, of course. Yeah. You know, you'll get around to it and you'll do that.
So, that was a big thing I learned as well. It's just, hey, some some some people, some areas, you have to actually be a little bit more proactive. And I do identify as a cultural thing because I'll have four or five people from that region, and they all react the same way. It's not like I'll have it from a different region.
So if you want, like, real specific region stuff, feel free to approach me after. I'm just gonna talk in gen genero generoities?
Now moving on to you.
So I'm gonna do one thing first, which is go back to the slide with all the peoples.
So this is the current state of our team. And as Nathan expressed to you, these were people hired as individuals as dots on a map. And one of our challenges is how do we get this to function as a cohesive business unit? Because to our sales team and our sales partner, we are a business unit that functions to serve them.
They don't know or care exactly how the sausage gets made. They expect us to serve them as a fully functioning unit of Upwork. And some of these contractors that we work with have other jobs, other responsibilities, other, you know, they are running a small business of sorts. And so I wanna just take a second to talk about the state of our org because our org is a lot like many of the orgs we've heard you talk about or I've had conversations with people about.
It is old and it is messy. As Nathan mentioned, we have a sales cloud instance. It was established in twenty eleven, and we've certainly had internal continuing to build it out. And our current strategy is just to start to do org hygiene one bite at a time.
We've got currently two hundred and fifty sales users that we support, Some of whom are engaged in that sales motion that Nathan described, serving our customers in terms of, you know, going out and getting actual new enterprise level fortune hundred clients. Others are, you know, we've got product data coming into the system and tickets and cases being open related to this work. So integrations are a huge part of what we manage. We also have a lot of custom development that is legacy and a lot of custom automations that we've built.
So our org hygiene is really, really an evolving process.
This is the current state of things today, where we're flowing things through a dev sandbox into a full sand that we really try to keep as a mirror of production. And what I'll say is that we're really good right now at continuous integration.
There's reasons that continuing development is a bit of of a struggle for us. Some of that we'll talk about in the next slides related to user support and communication, but it really is, something that is that is a constant state of evolution for us. We have a history before Nathan started of a two day SLA, and a lot of our developers still live under that mindset. It's ready, fire, aim. It's get code out the door as quickly as possible. But as you start to serve and try to build a global cohesive team, you've got to build team trust and communication into those channels and processes.
And again, historically, we're very driven by custom Apex. As we've built in more integrations, as we've added in more, you know, more apps, and even partners in the Salesforce ecosystem that develop products to our team that are flow based, the way that we view our code processes has really changed. And Gearset's has been huge for us because it lets us being able to develop and review things at the line of code level, which creates technical trust. And in distributed team, that technical trust is just really, really important for us.
So how we communicate both internally and how, you know, how those dots on the map come together and support each other as well as how we support our users.
Our processes here, we use very standard tools. We are Jira driven. We have a really old on prem edition of Jira that's not even on the cloud. It is a pain in the butt to work with.
It cannot integrate with a lot of things, but it's very real. And so we work to get our teams to communicate with our stakeholders through the ticketing process. Because sometimes because of that fear of failure, it's it's a challenge to get people to update tickets every day. We also use Confluence in terms of being able to pull things together and and use those those sprint tools in Confluence that are a little bit old.
We pull that custom HTML out of Confluence, and we drop it in a Slack channel for our leaders so they know what's coming in every week's deployment. We have daily stand ups as a team at nine thirty central time. It's the one time a day that all those dots on the map get together, and we talk in person, Monday through Thursday.
We've debated going to asynchronous. Sometimes we experiment with it, but our team likes that FaceTime together. They schedule individual calls after that to work on projects. It's that point of connection has become really critical to how we function, and it's how our team gets to know each other, which is a really huge part of, I think, our our success that we've had.
Our on demand support, we're really similar to a lot of folks, in terms of the tools that we use here. We have Haystack as our internal tool. We've built out a product knowledge base, and our line level support lives in the Slack channel. Our admin is in that channel. She can pull articles from the knowledge base to quickly answer questions, and uses that to develop tickets as needed to to escalate the support. We also have a lot of internal back channels that we're using for that asynchronous communication. Our folks in Pakistan may comment on those channels at different times of day and night, but there's a history and a place to go to and we've got the ability to integrate that into our ticketing system.
And then lastly, in terms of our actual sprint structure, we plan our sprints out three weeks in advance. We create a release note in calendar. And this is why continuous integration for us remains different than continuous development. Because we work so closely with our sales education team and user support, a page layout change that's very simple to push may require a video or an article or, you know, I've got a meeting Thursday to talk about turning page layout changes into little TikTok videos.
And I would not have thought that was something I would be doing a few years ago. But it is something that is very important to helping prepare for user based changes, which is something our technical users don't always understand. So being able to explain, this is simple and this gets slated for release in two weeks versus this is simple and it gets slated for release in two days is a constant struggle with our team in terms of how we talk about how things get prioritized. And it's why keeping our environments in constant sync is so important to a team that works like ours that has that end level user support.
So I'm gonna turn it back over to Nathan. We're gonna talk a little bit about some of our current problems and challenges and how we're using some of these tools to to work through it.
Yeah. So, so one of the things that we're trying to solve for, and I I help my team about this all the time, is, is is is is this balance between doing something fast and doing something well, and also doing things at a pace that that solution really needs to be done at. For instance, we had, this is a maybe it's a source object for us, but we had this field that we needed to disappear. Super simple.
Just remove it from the layout. It's all that needed to be done. But it got into our pipeline that was like our full sprint pipeline. So, I'm talking to our business partners, like, people are still asking what this field means and we don't need it on there.
Like, yes, I know. I I'm sorry. We'll we'll remove that. But it wasn't in a separate pipeline that was going fast.
It was in the old legacy. So it was like, this this is slated now to go release in a sprint two weeks from now. I understand that in five seconds, I could log in to the settings, I could click delete and it's gone. But two weeks is how you have to that's that's that wasn't exciting.
It's accessible to me, it was accessible to the partner. So how do we do these little changes, and we do them fast, and we do them, you know, efficiently, that that, you know, are causing problems, causing stuff, but aren't necessarily being subjected to these, like, you know, these longer, pipelines, these longer turnaround times that we're expecting for much larger changes that do need a bunch of training and roll up materials and stuff like that. So, for me, I think that's probably the biggest challenge I have is how do we do stuff? The slide says at the speed of business, versus, necessarily giving it the time it needs to incubate.
Because on a global team, things that have time to incubate are actually is really good.
Like, you can't do a global team with everything being a two day turnaround because you have to have the conversations, be able to work with the people, go through QA, do that stuff appropriately, but but finding that balance is always a challenge.
So when Nathan comes to me with a problem like this, there's there's a couple of things. Number one, it can be really difficult to get a very talented developer to care about removing a field. That's just a very real world problem. These are smart, talented individuals. They don't care. They wanna spend their time solving complex problems for business. So we wanna take the thinking out of that for them and use tools that support continuous integration so they can make that change quickly while they're doing something else and move on to it and have the right automations of pipeline that they do that once and we can carry it through quickly and effectively.
That that's a that's a very real problem to keeping people motivated that have the right skill set to do this. And and the other thing that we've been really, you know, fortunate that we've been able to do is cross train our admins in using some of these pro these tools and projects. So our wonderful admin in Mexico City can just make that change while she's doing something else and we have the ability to carry that through and solve that problem. We don't have problems like that that that we had once upon a time because we've been able to set up those environments and keep things in sync. Our devs can can use our branching strategy to resolve that. But it's a very it it was one of those things that those very small changes can be so visible to your partners, And they feel stronger and more confident about that same technical team because they can see those changes come through more quickly.
And so and now we're able to start to take that same pipeline, that same thinking, and apply it to more complex changes, being able to make adjustments to our flows faster. And and it you know, it's making the process work better, and it's increasing that stakeholder confidence, which is our goal at the end of the day.
So we will open this up for questions, interest you know, questions, thoughts, real world examples of where you've worked through similar situations.
Any questions? Yeah.
And, thanks.
And one of the challenges I think that I face as a manager is that we wanna do exactly what you're doing. It's like you have the big project work, but then you have those like small onesie, twosie type of requests that are simple, low hanging fruit, but they have to wait because of our, like, our processes, right, to be part of that two week deployment.
Trying to get away from that and trying to think of ways to creatively, like, make it all happen. So how do you support those changes? So, like, I guess, those, like, off cycle releases.
How do you support that with, like, code reviews and other quality checks along the way? Like, how does that all fit into the process to get delivered faster?
So the really the practical example is we still is we we use our ticketing system to create a record of that. And then we're able to make those changes quickly, run that through the backfill, and then it still gets checked as part of our weekly our QA analyst still checks those field changes and make sure they're still there, that everything's still working appropriately. And when he signs off on that week's release Mhmm. Those get validated appropriately.
We validate that there's in git, that everybody's backed everything up appropriately, that our devs are using the right pipelines for the sandbox, etcetera, etcetera. But it was true it was really a matter of making sure that we were following our own best practices and adding those one c two c changes into the sprint so that our devs knew they were there, they knew they needed to make sure that that field made it to their sandbox too, and and closing that communication loop. Got it. Cool.
Thank you.
Yep. Do you mind if you have time?
I I I think Just another one, yeah.
How do you what's the best way to put this?
How do you make sure that items that are probably larger than a small field change, but somebody still may feel like they need right now because it's mission critical. Who who's who's making that determination whether it's going speed of business or, you know, going through the the the full process, and where do you where do you kinda draw the line between those two things?
We run weekly sprint planning meetings and we created a field that we call an estimated release date that our stakeholders can look at and we say, look, this is this is based on today's knowledge. If we have to make changes to that field, we review those in our weekly sprint planning calls, and we let people know. This is something that happens for us very often. We had a very critical product rollout that was an all hands.
We're gonna stop doing everything else and everything got backed up. And this was about three weeks ago. So we had to say, hey, this is a really critical initiative. This is gonna get pushed back two weeks because of this business initiative, and and here's the visibility into what's happening.
What we've learned is that just being able to express that and giving people that estimation and understanding is much better than leaving them in the dark. And so just being able to point to something that they can trust and follow and document, here's what's happening, even if it's not even being able to show a developer action. This has been tested. This has been done even without commentary.
Seeing that action on the ticket and that movement has created trust with our stakeholders and helped improve that.
Yeah. For me, I would say it's it's authority and trust. Like, you have to have authority over the development taking place in your sandbox or in your work. When I first joined, literally, my boss said, two days.
Anything that comes in, any request, if it's a custom object, two days we're gonna do it. And I said, no. So that's not how we work. That's just that's not how like, it's great that you're, you know, gung ho and you're an operations person, you're not a developer.
And that's just we're not gonna be able to have the quality that we need and and work on a pipeline like that. So as as a you're telling me that I'm running this chip, this is this is what the expectation is. And so you have to have the authority to say that, and then you have to build trust where if something does get delayed beyond that time or something like like, you can actually communicate that ahead of time and people trust you when you say, this actually is gonna take me a week to do. Because I tell you this is gonna be done in a week, and I can deliver it in a week.
And I tell you it can't deliver two days, and it takes me four days even when I try really hard. Like like so so you you it it just takes consistency, I'd say, and and, you know, the authority and not trust.
I'd say we have time for one more question.
Yeah.
I'm just wondering why you guys made a decision to hire globally when your user base is local? Is that a cost thing?
Is that It's a few different things.
One, we are Upwork, so we do have access to our own platform. It is global.
The the mission statement of of Upwork is to kind of spread economic, opportunity to give people better lives, so that's our mission statement as well. And we, at our core believe that while businesses are concentrated, like, they they're in, you know, the major cities and urban areas, talent isn't. Talent is kind of anywhere and everywhere available. We've been able to find very good talent, very diverse talent that wouldn't have an opportunity otherwise if we hadn't kind of reached out and and offered them the employment just because they're not next to a big business center. So for us, it's actually, you know, in in line with our mission to work globally like that.
One question.
You say you didn't mention that you have a global team, right, and you do use pipelines.
So a developer does some development overnight, and if they have to deploy, somebody else carries the deployment between the pipelines?
Yes. So what we do is we use our partial sandbox as the catchall for a number of different things, and we're constantly backing things up into our git repo. And then our full sand isn't the true copy of production. And so we've got autumn like we've got automations and sinks running every few hours between several of these environments to keep the developer sandbox as current on the projects they're working on and then we use our staging sandbox truly as the wild what we call it the wild wild west internally.
That is where that is truly where if we're testing a new app, testing a new integration, code gets pushed there, and it's step one in the QA process. And it's and it's an imperfect process. We do you know, when we started this, we had people writing over, and and and that was a learning curve for us. So that's one of the reasons that we made the decision once we refreshed our full same environment most recently to keep it clean and pure.
I know we're running up on time. So Cool. Thank you, everyone. Thank you. Great discussion.
