Video: What WFM teams are building with AI: Vol. 1 | Duration: 3556s | Summary: What WFM teams are building with AI: Vol. 1 | Chapters: Welcome & Introduction (5.6s), Welcome and Introduction (132.85s), Key Terms Explained (302.995s), AI Role Framework (431.415s), Danielle's Introduction (552.01s), Forecast Analysis Pages (667.335s), Planning Assumptions & Forecasting (747.175s), Q&A Session (895.835s), Agentic RTA Analytics (1071.895s), System Design Implementation (1314.255s), Alert Workflow System (1547.945s), Deployment Crisis Recovery (1721.875s), Future Roadmap Planning (1849.825s), Q&A Session (1947.73s), Queue Management & Integration (2039.76s), Testing and Guardrails (2156.415s), Brex Presentation (2194.862s), UTO Bot Solution (2230.325s), Automation Evolution (2374.105s), Live Bot Demo (2520.585s), Future Roadmap (2888.57s), Whatnot Presentation Intro (2982.175s), QD Automation Demo (3013.15s), Results & Performance (3115.175s), Closing Remarks (3414.425s), Closing Remarks (3496.04s)
Transcript for "What WFM teams are building with AI: Vol. 1":
Hello, everyone. Welcome. We're gonna give folks a few minutes to join. But in the meantime, go ahead and tell us in the chat where you're watching from, and then give us one word to describe your week. I'm streaming from rainy Portland, Oregon, and my word for this week is caffeine. Got West Coast, East Coast, Caffeine plus one. Another Portlander. Thomas's word is busy. That's my second choice. Yes. I see a few busys. It's been a busy week, hasn't it? Alright. We'll just give one more minute, make sure people have a chance to join. Laura's word is snacks. Love that. Reminds me that I didn't eat breakfast yet. I'm hungry. Oh, no. Excedrin. Hopefully, that headache is clearing up. Okay. I think more folks will be joining, but, let's go ahead and get started. So welcome to the AI for WFM show and tell. I'm Whitney, and I am on the marketing team at Assembled. And I think for the last few years, everyone here has been hearing a lot about AI. But I think that that AI has mostly been customer facing. And now that chatbots and voice bots are kinda table stakes, we're starting to hear teams ask a different question, which is where does AI fit into the operation itself? There are more than 630 people who signed up for this webinar today, so it sounds like that's a question that a lot of you have as well. At Assembl, I would say that we're a team of builders. And, you know, over the past few years, every department across our org has been experimenting with how AI fits into and reshapes the work that we're doing. And one thing that we've learned as a team is that seeing what somebody else has built is the fastest way to get other people experimenting. So that's what we're gonna do here today. We've got four teams that are gonna show you what they have built. But first, I wanna get a sense for where are people here at in their AI journeys. So, we're gonna have a poll. I'd love for you to tell me, how is AI used in your work today? And then, you know, we'd also love to hear in the chat, what kind of experiments you're running. Or if you're not there yet, what are you curious to try? I also love that the chat is turning into a a reunion. That's awesome. Old friends, new friends. We'll give people just a couple more seconds to get their vote in. And, if they wanna share any of the things that they've been experimenting with or that they wanna experiment with. Hey, Adrianne in the chat. Alright. Thank you for your participation. Let's go ahead and close that, get back to our slides here. Okay. So wherever you landed, your results, what your team is doing in the poll... You know, where you were in the polls, here's what we're hoping to leave you with today. You're gonna see four real demos and what each one is made of and, hopefully, leave with an idea of where you wanna start doing your own experimentation and building. And, the friendly faces that we have on the slides here, I think it's important to call out that, like, these aren't engineers. They're your peers. And, you know, they looked at the work that was eating up their week, and they looked at the tools that they have available to them, and then they got curious. So, you know, I just say that to reinforce that this is all very accessible for, all of you who are early in your AI journey. Okay. So before we dive in, sorry. I'm having some technical difficulties. Okay. So before we dive in, there are three terms that you're gonna hear today. And so just wanna give everybody a quick primer on what they are. So an AI model is... You can think of it as, like, the engine behind tools like Claude and ChatGPT, the AI assistance that you use. And these models, they read requests, and then they work out a response. And then an API, which is short for application programming interface, is how software talks to other software. And it's really precise and reliable and usually set up with code. And then an MCP or model context protocol, it's a standard plug that lets AI look things up or take action in your tools, just by using plain language. So So that's when you're, like, in a cloud or a chat GPT. You're connected to, you know, one of the many tools that you use with an MCP server, and you're basically just asking questions trying to get the data and then, you know, in some cases have it take actions on your behalf. So, yeah, you're gonna hear that come up in today's demos. So, important just, one zero one to have, before we start showing the builds. And then another thing, you know, that I want you to be thinking about as we go through, these presentations is that, the builds that we're gonna show today, they showcase four different ways that AI can fit into the work. So in some cases, AI is the builder that helps you make the tool. In some cases, it's an analyst that's, you know, watching and and flagging what needs your attention. Sometimes it's an assistant that, like, goes as far as drafting changes for a person to approve. And sometimes it's an operator that's, making changes itself with guardrails, of course. So I would say, like, you know, there's no right or wrong answer or path to choose. I think it really just depends on what your team is comfortable with and, you know, how much, you know, how much comfort level you have with, like, handing AI the work, building with it, etcetera. So, hopefully, the the builds that we show you today will, get you inspired based on wherever you are in your in your journey. And then just a couple of quick notes before we start. So this session is being recorded, and we will email you the recording after the show. So, feel free to just skip the notes and, get in the moment and, enjoy the show. And then, there's also a q and a, tab. So, you know, you can drop your questions in there anytime. We're gonna, you know, go through each demo and then afterwards, we'll have a few minutes for q and a. We'll keep that presenter on and do some q and a with them. And then we'll save some time at the end for, group q and a. So with that, let's get into it, the moment that you've all been waiting for. First up, we have Danielle from FlexCar. So, Danielle, can you join me on stage? Hi, Danielle. Can you start can you start by telling us a bit about yourself and car and then show us what you built? Sure. So my name is Danielle. I have been working at Pluscar for four years now as their operations analyst, mainly in the customer care department. So pulling a lot of our voice of the customer insights and finding ways to, manage our, like, workforce management in our call center. We're a very small BPO, and we make sure that everyone is running efficiently, and Assemble has been, like, our premier platform in, like, managing our call center agents. Amazing. Alright. Well, I'm gonna go ahead and stop sharing and then let you take it from there. Alright. So today, I'm gonna showcase my labor planning capacity dashboard that I created, in Lovable with the assistance of the Assemble MCP. Assemble is our current forecasting source, and it's great for short and midterm forecasting. But it it doesn't quite provide as much insight into our long range capacity planning. And at Flex Car, we have a business forecast that lives and runs independently outside of our WFM tool. So it kinda makes it hard to truly, like, forecast volume and workload requirements based off of that expected member growth. So using the Assemble MCP has allowed us to integrate the siloed data sources into a source of truth. Pretty much to answer, like, how many people do we need to hire to keep up with demand and are we staffing efficiently? So just to jump right in, I'm going to show the dashboard that I created with Lovable. Alright. And so this dashboard has four pages to it. There's a phone and case forecast, as well as our member growth forecast and our contact rate page. The phone and case pages are pretty similar in nature, so I'll just show the phone one, and then I'll jump to member growth. On our phone and case pages, the focus area here is gonna be our dual track forecast versus actual chart. It has a statistical forecast that is a eight week look back of phone volume. And for our cases, it's a fourteen day look back because our case volume is very volatile at Flex Car with the amount of changes we have. It does run on a ninety day trend, and it does apply day of week seasonality. I also have, member driven forecast data as well. So we're taking our member growth volume that we forecast, and we multiply it by our two week contact rate, which we pull the amount of volume we have from the assemble NCP. And both of these forecasts can be, compared up against our Assemble forecast as well, and there is a toggle for that on this chart. And this really allows us to see, like, is the Assemble forecast matching up to what we anticipate with our member growth as well as our statistical forecast, and do we need to make any adjustments to the forecast that we have in Assemble. And then going to the top, I do have planning assumptions. So in assemble in the UI, we all know that we can adjust our AHT, our shrinkage, and our occupancy. But there are other factors in labor planning that aren't in Assemble, but we also wanna account for. So... Such as, like, attrition or scheduled slack. And these are some other factors that we manually were calculating when we were doing all of this planning and spreadsheets and compiling this data. And having these planning assumptions appear that I can toggle and adjust when we see certain trends increase or decrease has been really helpful in planning out FTE and how many people we need to staff. And then lastly, on the phone and case pages, we have, heat map gap analysis. Pull straight from the MCP. We have... It queries the average actual hourly volume per day, and it turns into hours required. And then we pull in the hours in adherence from the agent schedules to see where in our days that we are falling short as far as someone being available for our customers. And then jumping over to our member growth page, the main portion of this page is gonna be our forecasted weekly requirements chart. This is where our operational leadership is able to see everything in in one. So it can see our member volume. It can see how much, phone volume, case volume is forecasted. We're able to pull in our current staffing for our case queues as well as our phone queues from the assemble MCP. And we're actually able to see, like, the total staffing, what's required in that net difference. This has been able... This has been really helpful in helping us, like, plan ahead with hiring, which for us since we are such a very small team, it does come in very short cycles. We have a one month lead time to hire for people in our call center, and it takes about three weeks to put them into production. So staying ahead of that curve has been really important for us, and this planning allows us to do that. And so this is the, labor collection planning model that I built in Louisville. Amazing. Thank you so much for sharing. Danielle, we're going to keep you on for Q and A. So if you have questions in the audience, please drop them in the Q and A tab. I see, Dan, our friend Dan has a question for you and for all the presenters. But, Danielle, how long did it take you to build out what you've showed us here? Not long. It was a lot of iterations. I will say that. So I started off making this, based off of a spreadsheet. So all of this was spreadsheet based, and we had all the data we needed through Assemble as well as our own forecast here at FlexCar. I would say it took me about a week to build, and that is just with the many iterations of going through and making sure calculations on the back end are accurate. But building the base of it, maybe about two to three days, the full iteration of this about a week. Amazing. And then what lessons did you learn along the way? Definitely making sure that your data in Assemble is accurate. So your queues and the configuration of that is accurate as well as agent states and event types, because it plays a part in FTE. If we're pulling this data from the MCP and we're saying that someone is scheduled for phone and that's a productive event, we wanna make sure we're counting that. If those agent states don't align, then your data will be skewed. Definitely. Alright. Are there any more questions for Danielle? Oh, I do see more questions. Okay. Are these solutions dependent on your personal AI environment, or are they being developed and deployed through a shared infrastructure that allows others to maintain and expand them? So it is a shared infrastructure. I am the primary owner, but, our operational leaders are also able to access this as well. And how many people on your team are building AI tools? Is it just you, or is everybody experimenting? Everyone is experimenting. We are a very small team. It's only three of us, but everyone is experimenting. We also use Claude as well for a lot of our real time analysis work, and we're using and leveraging the MCP. And this dashboard also is connected to to Claude to, ask questions and query information. And then, last question for you, what are the direct tools or systems that are integrated with your dashboard? Is it only assembled or... a symbol, but it's mainly a symbol, but we also use cloud as well. Awesome. Okay. Cool. Well, Danielle, thank you so much. We'll bring you back up here in a little bit for group q and a. And next up, we have Christian from. Checkr. Yeah. Thank you. Alright. Christian, you're being called to the stage. Christian, can you tell us a bit about yourself and Checkr and then show us what you made? Sure. Nice to meet everyone that I can't see. My name is Christian. I'm from Nashville. Been at Checkr for just over a year. If you're not familiar with Checkr, we've historically been known as a background check company, but I would say we're, more in the space now of, empowering people to make good decisions for anything that they wanna do. I lead two distinct functions here at Checkr. I have a a tools and systems team that maintains any ops tool that our that our teams work on, both for human workers and for digital workers. And then on the flip side, which is more relevant to here, I also lead our our workforce management program as well. Amazing. And then I'll go ahead and give you the screen. Cool. K. Let's see here. Alright. Stupid things. Okay. So I'm here to talk a little bit about, an RTA analyst that we have been iterating on, for the last few months, ultimately leading towards some future vision where we're gonna have, like, a fully AgenTic, RTA, product built. So there's, like, a few different names that my team has gone through. They use Sentinel for a while. They use Quantix for a while, but generally speaking, we're talking about the idea of AgenTic RTA. What's really important to know and is, like, a big thing for our team culture is that there are a million different names that my team loves to use. They're all fun. They're all acronyms, and I remember zero of them. I may or may not have had them create a specific website that has every link in one place because I don't remember the names. They would probably be embarrassed that I would tell you that. But anyways, we're talking about Quantix today, but I think ultimately lends itself to an agentic RTA type future. So this is probably not a unique story for many of you, but, a key problem that we had was that RTA was being watched sometimes, but not all the time. You didn't have a twenty four seven operation for coverage for queues that do run twenty four seven. It was also really different depending on who was on the queue. I had a really big issue with consistency. Sometimes the same queue may have different, different messaging or what people thought was an issue, for some may not be an issue for others. Regardless of what an SOP tells you or what our, like, typical procedure would ideally be, obviously, humans can make mistakes. And then the the most most important part is that none of it was connected, the way that I wanted to. Right? We had our Salesforce queues doing some things. We had some Zendesk queues doing some things. But instead of it being a well oiled machine that was connected to every bit of the system, it's just stand alone, and we were just monitoring it like a hawk, across a number of different places. Ultimately, the the thing that matters most and, like, kind of was our headline of why we wanted to solve for this is. our target first response time for our messaging channels is ninety seconds. And if we see a pile up starting to happen within the first few seconds of, you know, an over index interval, the the the problem is gonna really hit fast and every second counts. From the moment that they start typing to the moment that an agent needs to get either move queues or we need to have somebody go online, whatever the case may be, the likelihood that we meet that first response time goal is, pretty low. Now, obviously, the goal matters, but for for us and what that, like, ultimately is grounded in and how I talk about it with my team is it also means that we're missing what the customer expects of us. Right? If they expect us to be there and we're not, that's a big issue. So, you know, talking a little bit about where we started when we had this problem and we've identified it, we do not have a specific, like, platform or engineering team that was staffed to this. We also kind of have a a fairly unique team design here at Checker. WFM is mostly outsourced under me, and we have a... I have staff augmentation. I have a... I have six people who sit in India through a vendor who do WFM, but they work on our checker systems. So no one was here that was, like, traditionally a builder, and at least in the engineering sense. We also... Similar to... I'm sure many of you, Claude is our multiplier. It did all of our coding. It did our research design, and many of the builds. So that was like a a key bit of how we were using AI. And then we wanted to start really, really small. I had a lot of conviction and talked a lot to my team about this especially in q three that a lot of the decisions we were making were fairly deterministic and it was a pretty good place to start. If we saw that our chat queue was getting close to our breach timers, we we know the decisions we're going to make whether that's hands on deck, we might call for overtime, we might people pull people off of other channels, we might people pull people off of other queues. There are a finite number of things that are going to happen depending on the scenario, but I wanted to start really small to make a deterministic decision. So basically, here's how it works today. There is a, timers that are happening across all of our Salesforce queues that are hard coded for the the specific threshold. So for us, that means that we might get to an email within the first twenty four hours. The chat is maybe gonna be ninety seconds, and then our phone calls are around three minutes. And the goal there is to not have a a... An alert set at the end. Right? If we're putting an alert based on eighty seconds for our messaging, we have ten seconds to make a reaction. So it's based on a mixture of both volume and time, and what it starts doing is posting a Slack alert, which you'll see in a minute. I'm using, like, an anonymized version of it to obviously hide our data, but generally speaking, it'll be the same. That Slack alert then gets posted to relevant RTA channels where, my team will then coordinate to get people in the right queues, channels, etcetera. Today, this is not running on any sort of, like, specific language model. It's fairly deterministic and is not making, like, decisions itself. It does, however, make some recommendations for moving agents, for, calling for overtime, or for an ex place of extreme health calls for b... VTO, etcetera. I think what's really important and something that we talked a lot about upfront before we got to, like, the the final implementation was building the right guardrails. So it is read only. It's not pulling anything from Salesforce. It also doesn't have the ability, I would say, yet, to change people's, aux codes. It can't requeue them to, like, change their actual assignment. Right? It's only reading the specific data. It is counts, only. We're only pulling the data. We don't pull in any specific, PII to the flow. And, frankly, I'm not sure that it'd even be necessary, but we aren't. It is based on human thresholds, so we hard coded a bunch of different thresholds with different scenarios to lead to different levels of escalation. And then again, ultimately, a human x we tested. And there's some things that we've defined as, like, off limits at least for now. We we don't recommend any, like, mandatory overtime. We would never frame it that way. We don't recommend cross training, and we... We'll never pull people from, like, actual training that they're already in. We use many other levers that we're okay to, like, to recommend based on the thing. So here's the basic flow. So, again, this is the the basic layout of what you'll see. So when a queue crosses its first threshold, let's pretend this longest wait time is forty second... Forty one seconds here. If nothing happens and it moves to say one and eleven seconds, it'll then it'll then say reminder attention. And this is showing the flow of a few things happening, again, mock data. It'll post it twice for the same issue, and this is actually a pretty mild example. Something that I like to see on the weekends when I'm not necessarily working is my Slack notifications, which I leave on. If for some reason something gets ignored, beyond, a healthy threshold... Let's pretend that, like, this one eleven... One minute eleven seconds, continues on. What's the right way to put this? I guess the Slack gets a little persistent, maybe a little bit annoying, and let's just say it kind of lights itself on fire until somebody pays attention and resolves the issue. But preventing it's less, less dramatic. Once this is given an app, once this is posted, then ideally, an RTA is gonna have people moved around. What then happens is we'll post a recovery, and the loop will close, talks about what what is remaining, if there's anything, that was missed. And one thing I really liked about this is if we talk about being answered outside of Target. I'm sure this is also not unique, but our data doesn't ingest in real time, so there's a lag to get to some of the data that I would ideally have same day. As a person who leads both the WFM team and a tools team, the question people ask me the most is why can't you make data available faster? And in partnership with my analytics, friends, we tell them that it doesn't adjust that way and that'd be very difficult and expensive. What this does do though, by saying it's answered outside of Target, is give us a really quick real time look to know how we're trending for the day and if we should course correct on the back half. And then the thing that I really do like, on the next the next morning, which is more, AgenTic, this is tied to Claude via scheduled task is each day there's an automated readout that comes based on all of the service level, whether it was healthy or not, and talks about the things that happen. It it not only looks at the specific queue health, which is important, but what it does is contextualize that data by pulling in, has access to our Slack instance, for example, if there were sevs, that would have caused an issue or let's pretend volume was just high and we hadn't identified it yet. It calls on anything that it can find that it believes is contextually accurate or will not call it out if it doesn't have a high level of confidence. A case study for this or like a a thing that actually happened, this was, my life a few weeks ago actually is... So, essentially, what happened was, on a Friday, which I don't like to deploy things on for my my tools friends out there, we did make a deploy, and it was near the end of day. And our phones are actually closed on the weekend where our messages are not. So we didn't catch this until Monday, but what essentially happened is our handoffs between our AI, upstream and then downstream into our human queues, broke. And Monday morning, our call, our our phone volume was, like, 10 x over forecast, and things were really, really happening very quickly. It was a bad Monday. But, you know, what could have happened in the past is maybe we catch it, maybe we don't, maybe people are paying attention to it, maybe people think it's anomaly, maybe it's connected, maybe it's not. Right? There's like this big if. What it ended up leading to is, you know, my alerts started popping off for multiple intervals. Sure. We got the right all hands on deck for the intervals. We started corralling some overtime, but then we swarmed it in a way that I felt was very different. So, you know, the RTA alerts flagged the problem to our humans. The RTA team then immediately looped in the dev team because they said this is above our thresholds. It's been consistent for multiple, multiple intervals, and it's time to do something different. Let's take a look. What we then found is based on the deploy on Friday, the hand off was broken. So what ended up happening is from the identification to the fix, it was twelve hours versus multiple days, which oftentimes I've seen pass. Ideally, we would have caught this on Friday, but, you know, that isn't always how it works. The one thing that I would say right now for us is that alerts are still not necessarily the full story. It's not a diagnosis. So it'll say that something's happening, but it's not necessarily saying yet why. And I think that changes. So to talk a little bit about what's next, the the rules run it today, but there's two distinct actions that I think come next. The first is moving us off of local machines. I saw a question in the first chat, which I... Or in the first section, which I loved. Today, this is on, my entire WFM team and myself's computers, but it's not on anybody else's. If we all won the lottery tomorrow, this would probably fall off a cliff and it would be, a big gap in the in the flow. So that is the immediate next step. We have to move off of local machines. And then secondly, I I think there's some work to do with a a Claude run layer that's gonna read the same similar... The re... The... Read the same signals, spot anomalies, and recommend different actions. And then ideally, stretching out a little bit further, maybe we'll use the Salesforce MCP, maybe we'll use the Zendesk MCP that's coming soon, and we will make immediate actions. If I know that I'm pulling somebody off of email because I need to put them on phone, I think that we can make that conf... That move with high confidence. Maybe we... I get... We alert the agent via Slack that, hey. This is happening. We're moving you. There's your notification. Then the system automatically does it, and we put them right where they are. And there's no real human in the loop there outside of, alerting when things don't happen the way they're supposed to. So maybe, like, guardrails or, you know, break thresholds, and then looking at the RCA the next, the next day. Few things that I think are important if you're early in your AI journey, start where you are. You, like, use AI to do what's boring, especially deterministic stuff is easy. Obviously, put your guardrails in place and then, you know, it doesn't have to be fully agentic on day one. I think building blocks are the most important thing here and the fastest way to get speed to value, frankly. Amazing. Christian, thank you so much for sharing. So jumping into the q and a here. How long did it take you to build this? It... The actual hands on keyboard time was maybe one to two weeks, but I will say that I ran the notifications and we tweaked some of the thresholds over the course of two weeks as well. So I'd say start to finish, we put this together in about a month and then ship to prod, in q four... Or excuse me, in q three. How. And what did you get wrong as you were building this out? did we get wrong? I would say what I... I was proven wrong. I thought that we would start with Adjentic, if I'm gonna be honest. I thought that the easy path here would be that we have enough decisions and enough understanding of our data and SOPs that we could pretty easily build the fully Agintiq version on day one. And we we... Maybe we could've, but I was proven that, proven wrong that that's not the fastest speed of value here. If we remove... If we get eyes on this faster and more consistently, we'll immediately start to see the value versus spending, you know, maybe four more weeks of build time. Maybe that's an exaggeration, but, doing nothing. So I think we just had the wrong... I had the wrong expectations maybe upfront of what was first. Yeah. I think it's, the phrase that I hear a lot and that I come back to a lot is that you're building the plane while flying it, so it's okay if it's not perfect before you take off. How many queues do you handle? And can you mention how many contacts you handle. per day? Queues. Let's see. So we run email we run email messaging, phone for at least two teams, so call it six there. And then there are... 1234567. Call it somewhere in the range of 12 to 15 queues, at least in my Salesforce instance. And then as far as volume, I would put us in the, like, 10 to 15,000, all in issues per week. And then another question for you. What what is all integrated to achieve what you've built? Today, it is... Let's see. It's running on local machines via some script. It is connected into our Salesforce instance via MCP. It does have some connection into the assembled MCP, although we're not using all the functionality yet. Trying to think. I think those are it. And then, someone here who said they're on the same path of building an RTA agent. Have you noticed any difference for the agent's response to an AI ping versus an analyst ping to move? My... So my... At least today, the build is not talking directly to the frontline workforce. It's either working with my RTA staff or, leadership at a BPO. I could see some hesitation in the future depending though on how we deploy, to to to how agents would respond. And then last question for you. What were the main guardrails that were put into place? The main one that I was concerned about is that we didn't act fast enough. So we tested what's the amount of time where we could actually inflect change very specifically for phone and messaging. That took a little finessing to get to the right amount of time to move somebody. And then the other guardrail that we talked a lot about was not annoying people via Slack too much, but annoying them the right amount. Yeah. Awesome. Well, thank you so much, Christian. We'll, we'll have you back up on for group Cool. Thanks. q and a soon. Alright. Traydon, your turn. You're up. Hi. Alright. So go ahead and tell us, a bit about yourself and Brex, and then show us what you made. Of course. My name is President Casper. I am the capacity planning lead, for the support team at Brex. And Brex is a spend management platform, putting, you know, corporate cards, expense management, budgets, and travel all into one system. And what we have built is... In fact, let me go ahead and share my screen, and we can kinda go from there. Setting up. Alright. Please let me know if you can see my screen. I see it. Perfect. Yeah. So what we built is is, a UTO bot, with assembled. And so just to start it off real quick, a lot of people might use different terms, but UTO is just unplanned sick time, or, you know, unscheduled time off within a twenty four hour time frame of when their next shift starts. So when we think about UTO, we're talking, anything that's, you know, the next day or, you know, it's happening within an hour. Anything past that twenty four hour mark is more PTO and and, is made through our PTO, process. This has been a huge issue for our team. You know, our team is split between BPO specialists as well as our internal specialists, and our WFM team handles the callouts of all of those, BPO and internal. And so we get several, a day, and it is a time consuming process for the team to, one, respond to them. When we get the call out in Slack. You know, they have to go in. They plot it into assembled. They logged it into an attendance tracker that we have, and then they reply by hand to, that Slack message. And then from there, they also need to, you know, recheck staffing for that specific queue during that day. Are there any breaks, lunches, or offline events that need to be rescheduled? And for the specialist that called out, are there any events that they are required to do, whether it be a training or, a, you know, one on one time that also needs to be scheduled? We need to go in and move those around as well. We've been looking at AI for quite a while now, and working through different ways of building this out. And our very first instance was using Glean, and creating our RT AI bot. And so for a specific, unplanned time off callout or sick callout, it it handled three tasks. It was able to plot the time off and assemble using the API. The... It added the attendance tracker log, which was to a Google Sheet, and then it was able to reply to the specialist in Slack. But it didn't really handle the more meaty part of that, this workflow, which is looking for the coverage gaps and rescheduling and making the changes within the, our assembled schedules. So it saved us, you know, about two to four minutes per UTO request, which across about 237 UTO requests a month, that's about eight hours. We recently moved on and and, got to our, GrokBot. We've we've used a lot of different AI things, chat, GPT, then we moved to Claude and recently got access to GrokBot. It it took us about three months to to get Glean up and running and and where we wanted it to be from an RTA I bought. That's not just, you know, UTO, but it... It's act... Actually responding to the specialists in our Slack channel. But for this specific instance, it took about seven and a half hours of communicating with the bot, building it out, testing it out, and, it's been, really awesome to see how authentic the the bot is able to run compared to Glean, which is more workflow based and, you know, reviewing epic queue docs that we have to state answers, and respond to the specialist. And so what the new flow saves using GrokBot is we are able to do the entire flow, from start to finish, completely automated, and that saves us about fifteen minutes to call out. And with the same 237 on average that we're seeing, that's almost sixty hours saved a month that our RTA team is now able to put into, you know, looking at trends for the day and making the right callouts to hit our service level targets. And so what we'll go ahead and do is I'll go ahead and and exit this and give you a quick demonstration of how we use it. And so I'll open this up, and we'll go ahead and move over to our Slack. This is a specific Slack channel that I used for the demo. As you can see here, my peer, Patrick, has also been assisting me with this. He's also in the backstage and will answer, some of the questions that might pop up. But we can make a simple thing here by saying... So saying I just need to take the rest of the day. It does take a a a brief moment to respond to us. And as we do, what I'll go ahead and show is just the demo schedules. Again, I've recreated this bot in more of a test, scenario. So in assembled, our schedules were showing up as test schedules. It's not really, you know, our actual team members, but the names here are just our our WFM team working at the time. So if we go in here, we can come into our Slack. One of the big caveats of Grok at the moment for our team is that, we don't have approval from legal yet to act... Have the bot respond, but it does respond as a WFM individual. And so we put clear indicators of when the bot is responding. So you, you can see the title of the bot, and you can see that it's being sent using cursor. But, you know, it sends the response. Thanks for reaching out. Working to update your schedule. Now you're all set to have the rest of the day off for Friday. If we go back in here and refresh the page, we'll be able to see that, UTO was applied for me for today. And then what we can see is if we go back to our, Slack channel, we'll be able to see that the bot has actually responded, giving coverage proposals and recommendations for the WFM team to approve. In a normal scenario, the first two pieces are going to be responses to the specialist, and then this third piece is going to be a response to our WFM team in our... In a different Slack room where it's given the suggestions of changes made, and giving us an an overview of our staffing for chat and, email as well as phone. And so we can then go in in here in here and say, you know, let's approve one, two, and three, and it'll start working to make those changes. You can also see Patrick is is gone and and had requested some sick time today for us as well. And if we go in here, you should be able to see responses. Updated. Let me go in here. And then... So what it does once the approval comes in, it's it's working on the back end to make those changes. Making those changes takes a little bit more time as it's, you know, going through and updating. One of the nice things here as well is that with this manual view, I can also say, you know, this training, you know, it's saying to cut out four. We can say approve four, but reschedule it for next week. And so we can see that it responded, made the changes, and was able to do everything. It did say, you know, four was not approved and it's unchanged. Phone stays short this afternoon even after these moves. You know, this is basing it off of our real, phone and chat. So while it shows this demo, it's it's looking at more than just the five agents. And with this, when we we say reschedule it for next week, it's looking at our staffing versus scheduled, required demand and rescheduling based off of that. So from there, everything gets updated. We're good to go. And if we go kind of back to, the slideshow, one of the honest lessons that we've learned, throughout this whole process over the years, make... Making sure you have checks and balances in place. We all like to imagine that AI is, basically able to do everything we do all at once, you know, with very limited, prompting. But what we've learned is that you do need to check-in on it, make sure it's not hallucinating or, putting in incorrect information. For example, earlier this year when we were using Glean, we had a rule in place that Glean was to grab the time zone of the specialist where they're requesting that time and convert it into, Pacific time, and apply the schedules since all schedules were based in Pacific time. And, unfortunately, we assumed that that was good to go and found later on that it had broken at one point and had about a week's worth of data with incorrect, you know, time off or UTO applied to the schedules, which we then had to manually fix, excuse me, go into the bot and, change that. And so had we had a text and balance in place to flag if if something looked off or or different, we would have been able to catch that quicker and, you know, make those changes and adjustments. But looking out to... For what's next, this this feature here is just one step of of the full, RTA iBot that we have, and we're in the process of migrating everything over to GrokBot to make it a little bit more agentic. So this will flow into the full process, and and we anticipate even more time savings with, getting more procedures added to this. Another one is also the assembled to Workday two day, two way integration. We're excited to be able to get that set up. And as part of the flow, submitting that UTO in assembled, which then update... Updates the specialist Workday time, reducing the need for for the specialist to, one, request or flag the time off, in Slack, and then also have to go into their workday and request that time off or remember to do that the next day. And then also just ensuring we have a good, tracker, and checks and balances in place. We also have a different bot that reviews the performance of this bot and flags any concerns. So AI working for AI. It's been, you know, a lot of fun, but, yeah, really excited about what we've got going on and and happy to, you know, stop sharing and, yeah, answer any questions. Amazing. Thank you so much for sharing that. I think, we'll do... If people can drop their questions in the chat and then answer there, we've got ten minutes left. So I wanna make sure that we're able to get to our next presenter. But, yeah, that was awesome, Traden. Thank you so much for sharing. And then now I would like to have, Beau and, his teammate Mark join us on stage. Why don't you guys go ahead and introduce yourselves to tell us about whatnot and then show us what you made? Yeah. Absolutely. My name is Beau Burton. I partner with the workforce management group here at Whatnot. I'm gonna move pretty quick. I've only got ten minutes left. But, basically, I joined the team eight months ago, and and the business problem in front of us was we had to build systems that could scale really fast. We needed a large number of capabilities, and we needed to be able to do that without having to hire a bunch of additional headcount, and being able to adapt to, the additional growth that was gonna come our way. So, we had recently onboarded assembled, and they have a great API infrastructure, and we can talk about how we leverage that. But we just started building, tools and and trying to figure out how much stuff we could safely automate. Fortunately, for us, and the group, we had Mark here. And what we're gonna demo is QD, which is, Mark's brain and energy here. But it has been great. We'll we'll jump into a demo. But before we do, Mark, anything you wanna share on on QD before we we get into it? Yeah. So it's, like, going to the build time and stuff afterwards if we have time. But, it just... There's a lot of tasks that we do every day. You know, a lot of it have been described here so far in the different presentations that... You know, just being blunt, I just don't really enjoy doing them, like the small tasks. So I just found a way. to try to automate it and get it done, find it, and, you know, make sure it's, like, in logic and there's a process for how it's done and all the guardrails, everything involved. And then is... You know, for this presentation, I thought what better way to show you that other than an AI video, but you'll see. Yeah. And if I can get... Here we go. Okay. Cool. Let me show you what we got. First, the problem. Schedule changes reach WFMS Slack posts. Callouts, late arrivals, meeting moves, and PTO. Someone reads each one, finds the agent in assembled, and types the change in by hand. Nights and weekends, nobody is scheduled for that work. So the requests wait for Monday. Until then, the schedule is wrong. Intraday staffing counts agents who aren't there, and adherence flags people who did what they were asked. Qd is a quad based bot that reads the schedule change channels in Slack at any hour. For each request, it works out the exact change and writes it to assembled through the API. A check means it's in. If something is missing like which agent, it asks the lead in the thread. The answer comes back to the same place. Anything past its limits goes to a person on the WFM team. Here's a real request from Dublin. At 10:13 in the morning there, California is asleep. The lead asks to move Laura's one on one to noon. Cutie waits a moment first since people often add a detail in a second message. Then it reads Laura's day through the API, her activities, and her time off, so it sees what's already booked. There's a break at noon, so the plan has two moves. The break goes to the nearest free spot at one after the listening circle. The meeting takes noon. Before anything is written, code checks the plan. It has to sit inside her shift in her time zone and knock nothing else off. Then it writes the break first, so the meeting has room. It reads the schedule back to confirm both moves landed. The reply in the thread says exactly what changed in Dublin time, three minutes post to reply. The same loop covers the harder cases. When a request clashes with something already booked, it asks before it changes anything. Here, the agent already has questions channel booked then. Night shifts cross midnight, and assembled splits them in two. Cutie still enters the whole night as one sick block. A weekly one on one goes on every Wednesday in the posted schedule. That's nineteen weeks, and it skips days off. Sometimes PTO is approved, but it never reaches the schedule. QD finds the approval and fills the shift. Its second job is intraday. At seven each morning, it grades every queue. Each grade compares the backlog with a safe line. The safe line is the most work the scheduled team can carry. Above it, new tickets miss goal. A queue over its line gets the hours needed to catch up. Here, HVI needs about four. Every four hours, it posts who should work which queue. The order comes from need, skills, and real time Zendesk status. Team leads copy it to their teams. And check ins every two hours show whether the plan is working. The same pattern runs other jobs. Before an operations change launches, like a new queue for VIP claims, it writes down what the change should do. After launch, it measures what landed against that plan, on a sheet WFM watches until it settles. And in the RTA tool, WFM and ops leads ask staffing questions in a chat. Can six seller agents train Thursday from one to three? The answer comes from the same queue model the tool shows. Yes. From three to five, seller would go over its line. How it's built? It starts with a Slack post, a form, or free text. Code turns it into a case. The thread, the agent schedule, time off, and related posts, all as text. That's what a person would look at. One Claude call decides, do it, ask, hand off, or nothing. Code checks the plan against hard limits, then runs it as a dry run. Only then does code make the API calls, read the result back, and reply. The model has no tools. It can't write to assembled, only code can. And each early mistake, like a time zone slip, became one more coded check. QD now runs all day. Time off and requests outside policies still go to people. Of all the other CX requests, it handles 95%. Handled means it entered them or asked the lead for what was missing. The rest go to a person. The median time from Slack post to assembled is one point eight minutes. And the model costs about 12¢ a decision. Alright. So you can see the range there of what QD is tackling. We really started with one problem, you know, can you report properly to... And can you make schedule changes? Can you see the volume? Right? And then just iterate it from there. A lot of use cases we have built aren't aren't in here. Mark continues to build these things day over day and and launch new stuff, which is crazy to see a scale. But, yeah, anything to add, Mark, before we turn it over to the group? I know couple of minutes left. Just kinda curious to other questions if there's anything to answer. But, yeah, I mean, there's just... There's there's a lot that's in here. I mean, since making that video, lots changed. I would say that one small thing in there about delegation seems like it's minor, but it's pretty... It's a pretty major thing, especially for holiday planning and, you know, bringing in, you know, specialty agents and BPO and, you know, there's there's a lot that goes into that. Yeah. Also, like, just wanna shout out the fact... Mark, you made that video with Google Slides. Is that what you said? No. It was, well, originally, it was through... It was an HTML, and it was, like, slides through an HTML. And then I just made it in... Well, I I made it into an m p four file. That's amazing. Yeah. It's still blowing my mind, and I've seen it a couple times now. Okay. Well, thank you guys so much for sharing. I think right now we're going to have to skip Q and A because our session ran long. But we're going to go and we're going to go ahead and have all of our presenters tell you how you can get in contact with them so that you can ask them any questions that you might have. And, yeah, just wanna say, like, a special thank you to all of our presenters and, of course, all of you for being here. And, yeah, I think, you know, I'm hoping that you're feeling inspired to go out and build. And when you do, get in touch with us because we'll be doing this again, in the new year. So, you know, you've got a couple months to go off and build something, and then maybe it'll be you up here demoing what you've built for the group. So, yeah, thank you again for joining us and, hope you got something out of it.