Name: Support Ticket Resolution
Uploaded: 2025-09-23T18:16:14.101Z
Duration: 59 min 4 s
Description: Support Ticket Resolution

Transcript for "Support Ticket Resolution": Hey, everyone. My name is Ken Lockett. I'm with, Celigo. I'm VP of product marketing here. I'm super excited to present, with a cast of experts, our support, ticket, resolution solution using AI. So I've got with me Julie and Sandeep and Yousef, and they'll all be joining me in a minute for us to go through, what we hope you'll find to be a very interesting way of leveraging AI in your workflows to, automate, or and do some deflection of support tickets. So with that, I'm going to invite Julie to join me, and we're gonna start with why did, why do we go after support tickets in the first place? Oh, thanks, Kim. So there are so many metrics that we use in support to really run the business, but here's three high level ones that I really keep an eye on daily that gives me a good insight across the entire organization. So first off, the quality of the interaction. So that has a direct linkage to customer satisfaction and ultimately our churn. So balancing quality with our other metrics is really critical to get a true holistic view of our customer's experience. MTTR is a common one. So it mean time to repair. It's really indicative of a customer's impact time. So the resolution times can really vary. The goals can vary based on the the complexity of the problem, but it's a metric that we really rely on that gives me insight into the team efficiency. And then the last one is capacity management or rather being effective with our capacity management. So ensuring our agents aren't overwhelmed, but also maintaining, you know, being an efficient resource allocating while maximizing their productivity. So we've you know, you have a large team of agents. You wanna make sure that the work is being shared properly across the right skill set, but you also wanna make sure that you're maximizing the productivity across your entire team. So pass it back to you, Ken. Awesome. Thank you. So we see a lot of people joining the the session here today, so we're excited to, see folks from kind of all over the country. If you've got any questions as we go through, please put them in the in the q and a. We've got a bunch of people ready to answer your questions, and we want this to be, you know, for a webinar as interactive as we as we possibly can. So Julie sort of set us up as to, like, why we won after support tickets. Right? We had some very specific goals in mind, and that's gonna be one of the key things that we wanna, you know, impart to you guys today is, like, you wanna go after things where you have very, very tangible and measurable goals, around the, AI solution. So when we sort of started looking at this, like, how do we wanna approach this? You know, we, I mean, we'll be honestly, we use Zendesk as our ticketing system. I'm sure a bunch of you out there use it as well. It's a great solution. So no disparaging, thoughts here. But what we saw in with the Zendesk AI that it was it was kinda locked into Zendesk. Right? It gave us a view of what was happening with the tickets that were registered in the Zendesk. And that that that was part of the the view that we needed. But we also had data, like, all other places. Like, we had data in, some Slack channels. We had data in Google Docs. We had, we had well, basically everywhere. We had some piece of data that would actually help our support agents, not only more quickly respond to inbound tickets, but, improve the quality of the response, especially as, you know, a lot of those, those agents sort of churn quite quickly. They move up into other roles in the organization, or they they move out. So what we were really looking for was not just kind of, you know, one, you know, looking at Slack AI, looking at Zendesk AI, looking at Salesforce AI, and then trying to store still stitch it all together to get an answer that support agent could use. We wanted to bring it all together into one, one LLM, and and, vector database so that we could then query that directly. Because if you could if you do it one by one, you end up with kind of a fragmented picture. Right? You you haven't really solved the problem. You've just added AI on on top of the existing problem. So that's where we started to look at our own solution, and and, you know, why we thought using our own integration platform, would be a useful, way to solve this problem. And, again, we we were very purposeful in how we approach, this particular use case. So we set expectations. Like, we thought that if we could bring all of that knowledge together into a central location, we would have, better quality, because we're using a broader knowledge set. Right? So we're bringing in all this information from all of our different sources to a single, vector database that we could then query. So we we made an assumption that that would improve the quality and the speed of the responses. So we also thought there would be sort of minimal risk to that as well because we're bringing in data that is already, been proven. Right? We're not making up information. We're not telling it to, to go do a task. We're we're telling it to do things that it's very good at, that that AI is very good at summarization, categorization. So really kind of the bread and butter where we know AI has some strengths, and we knew we could could control the level of hallucinations there because we're using our own internal data. And then lastly, we we had measurable, business benefits that we were targeting. And we knew that they if we could hit them, it would give us significant ROI on this particular project. So that's the prethinking. Right? Like and, again, I would highly encourage you that as you're looking at use cases that you that you do that. You're like, what are how do we measure this? What is the data that we're gonna pull in? Where does that data live? Is it AI ready? And then, you know, look for things that really will enable you to make a significant change at the lowest possible risk. So with that, I'm gonna invite, Yousef and Sandeep to join me, and we're gonna get into the weeds here. So they're gonna walk you through the solution and what we did, and go through a demo. So welcome, Yousef and Sandeep. Thank you, Kim. Thank you everyone for joining us in this session. Quick introduction. I my my name is Yousef. I'm one of the solution consultants here at Celigo. So and I have with me, Sandeep. So Sandeep, you want to introduce yourself? Hi, everyone. My name is Sandeep. I lead the AI, at Celigo where we deal with, billing AI for internal efficiencies and, business processes using our own product and LEM technologies. Yes. Absolutely. We, using our products, we want to, to basically, enhance our productivity internally, do a lot of things that we can do, usually, but this time with with the help of Gen AI. So, without further ado, so let's step in into, like, the our today's, use case, which is which is like the support, support use case, assistance. So, Sandeep, what can you tell us about about this here? So, it seems like the we have, like, a first step, which is the ingestion of the data. What are the systems that we are talking about here, and how we build this, for these systems? Sure, Youssef. So, in general, LLM technologies are, trained on broad set of data. So they are not, purpose built based on domain specific or business specific knowledge. So here, instead of relying purely on LLM, so we went with a RAG based approach, the retrieval augmented generation standard one of the standard patterns in LMM technologies. So what that, technology means is that instead of relying on LMM data, you give your own data, which is a source of truth. And then based on the data, you try to help users, get their queries. So that's the pattern we have, used here, the r g pattern. And, coming to the sources, right, like, let's say, the goal here is to help, a support rep resolve the ticket faster so that our MTTR and all of that, metrics come down faster. So we thought of, trying to see, like, what could be the sources that could help, support reps solve the ticket faster. So one thing is, the Zendesk tickets that are already solved because with our guard, product, right, it's, most some of the some of the times, the tickets are repetitive where we already, are close enough or those patterns are solved, same thing. So the Zendesk closed tickets, which are resolved, are one of good source for us to, find information to help, the sales rep. So that's one source we have. The second source is, support team. They have their own playbooks. Like, whenever they close a ticket, they have, documented steps as to how did they approach the problem, what was the resolution, whom did they talk to, what are the resource they help in. So they have data detailed their own internal documentation stored in Google Drive and Confluence. So we use that also as one of the source, for the information. The third, source we have is Celigo Connective. So Celigo Connective is a platform where our partners, customers, internal employees, everyone can collaborate on how to use the product, the issues they are facing, the best practices. So that's the forum where we have, engagement between different folks. So people post, issues. People post best practices. So we take that also as one of the source of, information to feed the agent. The next one we have is, internal Slack threads. There is lot of, conversations going on between, customers, customer facing teams like support, CS, and then engineering, and then l three, support team and all that stuff. So all that wealth of information that's hidden in Slack, conversations, we do take those, and then we use that also as a source of code when we try to answer, a query from the rep. So the last one is we have product documentation. So our product is very huge, and then we have a ton of documentation to thousands of articles written in Zendesk. So we use that also as one source of information because people could have questions, around general product and all. So these five sources are, where we where we rely on to answers, a support rep or help support rep whenever a new ticket comes in. Yeah. And so this is very insightful. And I think I think the main idea here is that we look for the knowledge internally in, in our case here, Celigo, whatever that knowledge is. So whether it's, an internal, messaging, tool like Slack or some documentation or community, and we get that knowledge. And, I think that the step that we see here is that we we use Celigo to basically, orchestrate between the OpenAI and, a knowledge base here. Can you tell us a little bit more about, what we are using here as vector database? And what is a vector database, by the way? So, basically, elements, we deal with, semantic searches, so similarities and all. So that's different with the, traditional syntactical search. So we use, Pinecone as a vector database, and then we use OpenAI, text three small latest embedding. So what we do is all the data that we have, we create embeddings for them, and then we store it in, the Pinecone for later searches. So whenever a user, comes with a query, so we take the query, we convert that query into embeddings, and then we go back into, Pinecone to say, okay. These are my embeddings or the user query. Get me the relevant articles that are close to this query. So now with from Pinecone, we get the list of articles, and then we have this confidence scores. Like, Pinecone gives us confidence score of 75, eighty, eighty five, and all. So we have a threshold of 85, as 85% or point eight five number as the threshold. So any article that's matching beyond, that number, we take that as relevant article to serve the user query. And once we have these articles, we ask again, OpenAI okay. You have this user query. You have these matching articles that are that can serve this query. So now try to help me, frame an answer to the question. So that's the pattern we have, Yousef, here. Yes. Yes. That that's great. And just for, for our, audience today, so embeddings is like for those who are not very familiar with the term, embeddings is a representation of a text as a vector. So it's a vector of, like, usually with OpenAI, it's like, I think 1,532 dimensions, with, like, numbers and it's representing a text. It's basically you are representing a huge text as a small vector with, a fixed number of dimensions. And we take that vector and we store it in a vector database with our text that we got from our knowledge, source. So, yeah, Citibank, we can see here that Citibank is playing that orchestration needed between the different the different endpoints, whether it's, OpenAI. And by the way, OpenAI, all the all the logos and the application that you see here, we are, really application and system agnostic. So we you you can use OpenAI, you can use Anthropic, you can use Gemini if if you are if you like, your company is, like, using Gemini as, your main LLM. And for for the, for the knowledge base and the vector database, you can use Pinecone or you can use something like PostgreSQL with, there is a plug in, VectorPG, where you can basically you transform your, PostgreSQL into a vector database. Okay. So can you can you tell us more about so this exact support Copilot agent here, what are we going to do with the next? So now so we saw the first step is basically we ingest that data into the vector database. Now what what next? So the next process, like, helping the rep. Right? So let's say, whenever a new support ticket comes in. So we have, two interfaces for this Copilot, either using Slack or Zendesk, custom app. So we have these two interfaces to interact, with the bot. So the moment a new support ticket comes in, we have a Zendesk custom app built in. So from Zendesk, we receive a query, to an IO flow. So all of these are powered by IO flows. In the back end, we have, an IO listener listening to queries from, Zendesk or Slack. So the moment we receive a query, now we do the extraction part. So we create the embeddings for that user query, and then we search in Pinecone, and then we give the articles. We give it to OpenAI to answer the question, and that answers goes back into, Slack and Zendesk. So the first, few things that could help us like, let's say, the moment a ticket comes in, we go through all the sources, and then we help them saying, okay. This is the, ticket that came in, and this is a possible resolution based on my understanding or based on the knowledge I have. So these are the possible resolution steps. And if you want to go deep into it, you can go check these these links, these Slack tips, these email, not even the, confluence pages or, other, community posts, etcetera. So the user has, almost if, got complete information context without having to search anything, everything is given to him. And then once he has that context, either he can try to fix the issue or ask more questions or ask as a copilot to draft an email or any other product related questions. All these things can be done here. Yeah. Yeah. So, that that that's great. So means that we are using to basically the the Zendesk, bot and Slack as our front end to all the whole system. Right? Yes. Great. So, let's let's maybe, take a look at the the demo right now, and maybe we can take a look deep dive into the platform and see how this is really built as, as, like, flows and, in our platform. And by the way, please, if you have questions, do not hesitate to put those questions in q and, q and a section, and, we will try to answer all these, your questions and, provide you more details about how this is, working. Okay? I'll stop sharing. Yep. Youssef, can you confirm if you can see my screen? Yes. We can see the screen with everyone. So Okay. So, yeah, this is our production. We see we see the the Comcast, the Goldcast, screen. We we don't see your screen. Can you see the I o flows, man? Yes. We can see your I o flows. Okay. So this is our own, production instance, of Celigo account. And then, so this is Celigo IO projects, our integration where we maintain all the integrations. So the support ticket, assistant is what we'll be going deep. But, beyond that, we have, a ton of, a base indications built using our own product, be it the product KB or the HR bot. And then we have ton of Gong related, insights, and we have few other things around request three sixty around customers data, customer case studies, and all of that. So So basically, I think I think every one of these group, flows that you are showing on the left side are like agents, doing something specific for us, in Cinco. Yes. Correct. Exactly. Sounds good. Okay. So, I think you, you clicked on that support ticket assistance, and we see three flows. So we the like, few minutes ago, we you mentioned, the first step which is ingestion, ingestion of the data. What are the flows that that you are using to ingest the the data here? Yeah. Here, we have two edition sources. One is that supporting, runbooks that they maintain. The second one is the Zendesk, close and resolve tickets. So, yeah, we will get into the the next one. Yeah. Can you go back can you go back for a second? I see that so the schedule there that for the flows. So two of them are, scheduled and one is real time. Can you give us, like, a little bit more details why there is a difference there? So, basically, these are like, the ingestion sources are running on schedule, like, daily ones to continuously update the data because, it's not a one time source that you give it some data and then it runs on its own. So as in when the tickets are coming in daily and then they're getting resolved, so the newer information is getting ingested into the bot for its, training and intelligence. Okay. So this one is a webhook, that's listening to the event. Yeah. Great. So it means that, when you are ingesting the the the the knowledge from the source, you don't have to ingest it in a real time, way. So you can basically schedule that to run, like, once a day or twice a day to grab whatever new knowledge you have and and add it to your knowledge base. Right? Yes. Correct. Perfect. Sounds good. So depending on how frequently, the data is changing, like, like, the support tickets, it could be a more frequent thing, like, every 15 minutes or every 30 minutes. Whereas, the product articles and all could be a deleting because the frequency at which they are updating, is not that, high. So yeah. Sounds great. Okay. So now we are looking at the ingestion, the the the tickets, the historical data from the tickets that were, in Zendesk. So are we pulling all the tech tickets, Sandeep? No. We only pull, the resolved tickets, like, which the support team have closed the ticket saying that, there's a properly resolved ticket. There could be closed tickets where the customer does not did not respond or not, so we ignore those. We only pick the right ones, which are resolved by the team. Sounds great. Okay. And so the next step is, once you get those tickets, what is the next step there? Yeah. So the first couple that we have here, it it pulls out, all the closed and resolved tickets. And, going further here, what happens is, customers sometimes just, send longer emails or, just copy paste, messages from other systems or not. So the ticket descriptions sometimes could go to the extreme. So what we do is, like, we have a filter where if the lengths are bigger than some specified, number, we summarize the text and then move forward, for ingestion because we do we want, crux of, whatever the user is submitting, to be indexed on the description. This is to optimize our flow. Right? So, keep the just the the main things that we want to keep in in our knowledge base. Right? Yes. Exactly. Uh-huh. And then next step, we get all the comments. Like, for a given, ticket resolved ticket, there could be a lot of two and fours between the customer and the support rate, like, 30 conversations, 50 conversations, and all. So we try to get all the comments, in this time because that's where the conversation is happening on how to resolve a ticket. So we get all the tickets. And then, we have a small, cleansing process because what happens is lot of times this, boilerplate text, in the comments because of email signatures from the customer or the rep email greetings and all of, these boilerplate stuff is not needed to actually ingest or get the course of the issue. So we have some cleansing process to remove that, remove some HTML tags and all. So basic basic data cleansing process happens in this hook before we start, getting the condition and all. Yeah. Yeah. So we are preparing the the data to be embedded. We don't want to embed, like, emails and, like, HTML parts. So we want just the content. And this is where this is I I think that this is a very good point because the platform offers a lot of, features and tools in that flow builder where it gives you the possibility to, to do the data transformation, apply some, advanced logic by, like, maybe using a script where you do some, something really, advanced when it comes to the transformation needed before you get to the point where you, you need to embed. And this is I think it's very powerful point, since, it makes the the this solution very customizable, instead of, like, just using the pre pre the the the solution that is really offered by, the different vendors out there that is not really, that that does not offer the same level of customization. Okay. So what is the next we embed the data. Yeah. So once we have the clean up data, so we create the embeddings, for those using the OpenAI model. Mhmm. And then, in the last, bubble, we push the data into vector database Pinecone along with that embeddings and the metadata required, for us to retrieve and add. So all that, cross happens in this last bubble where we ingest into Pinecone. Yeah. That that's great. Not sure Kim, if you have do you have anything to add here? No. I just, I just missed you guys. I was very intrigued by what you were doing, so I wanted to join the party. Sounds good. I think maybe, we saw a lot here. So I maybe you could take thirty seconds. You said first hand deep and just, like, summarize, because this is one part of it. Right? The ingestion piece, and then we're gonna go into the into the bot itself. Is that right? Yeah. Yes. So, basically, what we did here is we we ingested, the data coming from our Zendesk, tickets, and we we created the embeddings to build our knowledge base. That's the main idea that we have here. So if we go to the second flow, I think we have different, things that we can do there. We are pulling the data this time. What is the source that we have there, Sandeep? Confluence. We have Confluence. So yeah. So the customer experience engineering team, they have there's the l three support team. So they have, their own playbook stored within Confluence. So whenever they solve a ticket, so they go to the Confluence. They write a summary of how this resolved the ticket and all. So that's one good source of information for us to, feed for the next upcoming ticket. So here, what we do is we go through the, confluence pages. Like, if you see, we have a, search in the confluence with this label, see runbooks, so they label it. And then we only pick the modified, like, whatever, modified docs are from the last time the flow has run. So we pick those documents, for ingestion. So this bubble gives us the list of page, IDs. And, this second bubble will get the actual contents within a page. Like, k. And then, so here's some customization. Like, within one page, they could have multiple tickets. So the way they organize the page, right, they have product level, and within product, they have, the platform and IS. And within IS, they have multiple IS. And for each IA, logically, they have modular breakdowns and all. So we have some open AI bubble to, read the doc page and then, get multiple tickets information from the page. And once we have that, again, we have basic cleaning process here because, the conference is in HTML format, so there are a lot of tags, unnecessary things that we don't need to ingest. So basic, cleaning process, of the data. And then we create the embeddings for those, documents here in this step, and then we push it to confluence. So same pattern like the previous one. It's just the source is different, yeah, and running on a continuous basis. So the moment they add a new article, so this flow is running on a cadence, daily basis. So whenever they add new, pages, so that information is ingested into, the bot's knowledge on a continual basis. Yeah. Perfect. So I think the main the the new idea that we have in this flow compared to the other one is that we are using the AI to chunk or to, split the the, the the content based on specific, like, semantic, content. Right? So this is that step break single confluence documents into small, documents. I think in AI, we call that chunking, the the text. Yep. Okay. So, now we've we've seen, like, a couple of, are are these all the only ingestion, flows that you are using for, for the support use case? No. We have we have three more product. They are in different folders here. Mhmm. Yeah. We have three more sources, product, community, and, we have the Slack threads. So they're in different part of other use cases. Right? AI use case. Okay. Sounds good. So let's go to the main flow now where we are replying to the the queries from the the user. So the user will, can can you maybe let's before we we take a look at this one, maybe we can jump to Zendesk, and and see how that that chatbot look like and what is going to happen once you post, like, a question there. Okay. So, can you see the We can send us if you are sharing it. Okay. Can you see my Zendesk screen? Yes. Yes. We can. Okay. So this is our Zendesk instance, and, this is a ticket. So on the right side, we have custom different, widgets. So we built a custom, app within Zendesk, that serves the bot purpose. So whenever the first time the support rep, opens the ticket, so on the load of this bot, we make a call to I o, or, the bot. So and the bot responds bot takes in some information about this ticket description and all that, and then it responds in this way. So it gives you solution based on whatever is the issue. It'll try to summarize the issue. Okay? And then once you go down, it will give you references also. Like, let's say, these are the existing Zendesk tickets that are very close to, resolved tickets that are very close to this ticket, so you may look it as a reference. These are the list of three Slack queries that, are related to this ticket that can help you resolve, and these are internal conversations, happening. Similarly, the KB articles, these are product articles, that can help you, resolve this thing. Yeah. Yeah. Same with c e. So all the different sources, that can help resolve the ticket is here. And then there's a high level summary of the how do you resolve this. There's a, summary of the resolution also mentioned in the ticket here. Yeah. This is very, very interesting. So if I summarize this, so if a support rep is so once a support ticket is created, by default, once it's created, we this bot is going to, first of all, summarize the issue, try to find what is the solution, and provide the reference. So once this ticket will be assigned to a support rep, that support rep will be equipped from minute one to answer the question because that support rep will have in that widget, the application, the Celigo AI Zendesk chatbot will have already all the information needed. And if they need to check some documentation related to this, they can just click on, the KB articles. They can look at the runbook runbooks, previous ones, and they can answer start answering immediately the the question. I think I'm not sure. I mean, I I had, like, previously some support roles, in my previous, career. I think one of the things that we're we we used to do is basically you need to go try to understand and do some manual research into the knowledge base. Go over and try to find with, like, keywords what are the support tickets that are kind of, like, similar to the the one that you are trying to solve. And, and just look at the documentation, and it all this takes time. So it's, like, thirty minutes maybe just to to have the the the context before even starting to respond. Now it's like in few minutes, once the support rep is assigned, they can get all this information, available to them. Okay. So is this all what we can do or can we, like, ask and then request more information? Yeah. Yeah. We can request more, Yousef. Let's say, the second one I have. Right? Let's say, as a rep, I went through this thing, and I want to understand, before putting any, resolution to the customer, I want to understand what are the best practice around Huddl, shops and Huddl refund processing example. So now it it will give me what based on the documentation that we have, what are the best practice around how do we deal with that, and then references around where these are discussed, etcetera. Yeah. And then other things could be, like, let's say, you could ask you ask you to draft an email to the customer on how to fix the issue from their end. So, yeah, it will give you a rough draft of, whatever is it thinks, based on the knowledge, how can we resolve it. It gives you a draft of it. So you can take the draft, use your best judgment to see what is happening, and then make the changes and then send it over to the customer. Great. So just for for our, like, audience today, this is this is really, really interesting because now so, I mean, everyone has Touchpity. Right? They can go and ask TouchyPT to write an email, but TouchyPT does not know your your your actual data. They does not have access to your knowledge base, to your support tickets. When by using the chatbot and requesting, like, the chatbot, to, to write an email, to answer this question, you are using actually your proprietary, data, your private data, the organization's data without exposing that data, outside of, your organization. And you get like, you see there, it's it's basically providing some, an email, but also referring to the documentation, in our, Citibos documentation official documentation. Okay. Anything else that we can do with the chatbot, Sandeep? Yeah. Anything, around product question. Right? Let's say, as I read, I want to understand some specific functionality of how this works and all. So, I I can ask any question around how the product works. And based on the product knowledge base, we could answer these things. So, this can do pretty much anything, any question it can answer, as long as we have the underlying data source that could answer it. So yeah. Great. Okay. This is perfect. Let's now jump to the flow to see how this is possible. Right? Yeah. So before we go into the flow, one small thing, Yousef. Look, we have, Slack as also as one, integrated here. So people can come in and then just drop in a support ticket, and then we have a similar response, that happens on the Slack. So Okay. Yeah. We have multiple interfaces for the same bot because, only few folks supporting only has access to it. But, rest of the studio who doesn't have access to Zendesk can still use this bot, from a Slack context. Yeah. So, it's this is, again, another, very, very, interesting idea. It's that, whenever you are building the platform is you can think of it as your back end. Right? And the front end can can be you can choose whatever front end you want to use. You just need to have things like, a front end that can send a webhook request to, to some URL. Right? So, whether it's Slack, because Slack offers that, or chatbot in Zendesk, or even, like, something, that you can do, with, like, another chatbot available in the web web developments community. Yeah. Okay. We are looking can yes. So this is the flow. Yeah. Yeah. What what goes through through this, Sunday? So we have a listener here. So Mhmm. Any change, under Zendesk, CapReq, people asking query at the first time note. So we have a request coming to, the IO. We have a good book listener here. So the moment we get a, listener here so we have an OpenAI bubble first, input moderation just to ensure that, people are asking the right questions relevant to this bot, and this bot is responding only to the things that are they're supposed to do it, not just random things. So we have, an OpenAI based check here. And based on this OpenAI Just regarding this, Sandeep, I think I think I shared with you this morning an example. So a couple of examples, one from, Chipotle and, and and, another one from Bing, Microsoft Bing, where their chatbots exposed some data that was not supposed to answer. And this is the first step that you have seen this flow is what we call, an input validation. So you don't want to process something that is not related to the use case, specific to this, AI chatbot. So if someone is asking about, let's say, something related to the, like, the prompt, for example. Give me the prompt used for this chatbot. You don't want to answer that question. And this is why we have that first step there. So, yeah, once we have the safety check done, so we if it is not supposed to answer, we'll halt the flow. If not, we'll proceed further. So within the flow, if it's a new ticket, we get the ticket details. If it's already an existing ticket, we already have that, information, so we don't do that. So what we do is based on the user query, we create the embeddings. K? Once we create the embeddings, we search for the Zendesk closed tickets. Like, that's the first source. So as we scroll to the right, the flow is a little linear because we have five sources where we search information. So we will have, two bubbles for each one because we need to create embeddings for each, and then we need to search for each of them. So we have a lengthy flow here, but the code is that, like, we create the embeddings, and then we search the Pinecone, with that, embeddings to get the relevant articles. So once we get the relevant articles, I'll just move to the. Site. We also get the Zendesk ticket. And then here, we have an open air interaction where we are, giving all the information, like the user query and the articles from different, data sources, and then we ask it to give give the answer. So now, let's say, the top five articles from each sources, like, these 25 articles, five sources, five articles for each, 25 sources, and the user query is fed to OpenAI here. We use OpenAI threads, responses API assistance API. And then, we run the thread. We get the response. So once we get the response from OpenAI here, we have some post response guardrails also. This means that, we have another LLM as a validation to see if it's responding correctly or not. This under safety check, that we don't, put any harmful content or anything to the user. So we just have another, check before we send the content back to Zendesk. So after this guardrail, we push it back to the Zendesk app, and then we have some metadata tracking to Snowflake. So yeah. This is how the, retrieval, happens. Yeah. Yeah. So the main idea here again is is quadrails, whether it's an input validation or the what we we call an AI judge at the end to judge if that response is something helpful or not by exposing some PII or anything that is not supposed to be sent, before sending it. We are using at the end, Snowflake just from the metadata, but the the the the the main idea here is that you can custom and, your the logic that you want to build the way you want it. So, here we are pulling the data from a couple of, sources, doing the research. You can you can add you can create, like, more more guardrails, if needed, for for your use case. Okay. Do we have okay. So I think, next I think we do we have something else to show or maybe I think Ken? Yeah. No. That was awesome. Obviously, went through a lot there. I think you were touching on something that was interesting, Yousef, towards the end, in the last, in the last section there because I I that I wanna make sure that we do bring to the to this audience is, you know, really about the guardrails. Right? Because that's that's where the strength of of the strength and the weakness of any AI bot that you, create will will come to bear. So you you showed a little bit around, you know, where it's obviously anchored in our data, so it's not, you know, going off and looking at at, external sources. But, and this is to either you said first, Andy, what what other kind of, you know, typical guardrails are we putting into the prompts to make sure that, you know, we're minimizing hallucinations and avoiding responding to questions that, you know, are are inappropriate? Like, how do you kind of figure that that out? Yeah. Like, we still Good question. Yeah. Sandeep, do you want to, answer this one? Yeah. So, one thing is, like, PA and security information. So we have validations to ensure that, we are not exposing, anything that's not, that should not be exposed to the user. And the second thing is, from a best practice standpoint, we always try to, ground the a response on the data that we give between give it to it so that it doesn't hallucinate and all. So the smaller, the context to the a window, the better the responses are. And, yeah, we have, some process built built around, we treat prompts as code. So we have prompt review process also in place where, we have multiple pairs of eyes looking into, that we don't, do anything, off guard there. Yeah. I think these are the three things that, in general, we follow for everything. Yeah. Yeah. You said you can add the face in. Yeah. I I can add yeah. I can add in the prompt, you you want something that prevents you the prompt prompt injection, type of attacks where, someone is asking about the prompt itself. Obviously, we talked about the input validation, the output validation. So the I think the platform, the way you you have, like, all these tools that can allow you to do the transformation of the data and and and build this type of things makes it, like, easy to, apply the security, best practices when it comes to building this AI chatbots. Yeah. I mean, that's that's that's what I think is necessary. Yeah. No. I appreciate that, because I think that's, that's something people often sort of don't think about. Right? Like, people are even if they're just playing around, right, they're they're gonna try to break your your prompt. I know I've tried to do that a few times, but I've failed so far. So maybe going along those same lines around, you know, so you built your prompt. You you we we sort of approach it like a engineering, task. So we have got people reviewing it, but that sort of implies that there's a testing piece to it as well. So, like, how do we how do we test or how have we tested all of the the bots that we've created before we've, let them loose onto the larger Celigo community? So from a practice standpoint, Kim, like, once from a development standpoint, once we do, right, we have, developer review process and all that's closed. So before we bring it to broader broader audience, we have a pilot team. Like, so in this case, we had, I think, five or six support reps that we've worked with where we had closed, meetings with them, tried to see how they use the product like, and then we have seen the responses. The bot is, giving an alert. We took the feedback from the rep. It's useful, not useful, what to change, the tone of message, the, structure of the message that we have and all. So once, we had that, pilot group of, reps worked out and tested, And then we got that feedback. We went to broader audience. And then once we go live to the broader audience, we have, strong reporting and monitoring in place. Like, we have, as we have shown. Right? We have Snowflake where we are showing the data, so we have thought spot reports to see, how is it being used. And then we have a feedback capture mechanism also. Like, people can give feedback useful, not useful within Slack and, Zendesk. So we capture that into and then we go back to them and asking, what's working well, what's not working well, and then we try to make it a continuous improvement team. Awesome. Those those are my thought. We're good. Best practices. So I'm gonna thank you guys for, imparting all of that information going, into the weeds about how we actually built it. I'm going to invite Julie to come back, because, obviously, we've done all this work. We've deployed the, the bot, and, we have seen we've had it out for, I wanna say, eight months or so. So we've been looking at, our metrics and have been, looking for, results. So, Julie, let's can you walk us through the results that we've seen so far? Sure thing. Yes. Actually, happy to do this. So, we've got some early results from the support team. Again, we've been using these bots for about eight months. So we've increased our throughput without really putting any extra burden on the team. So our tickets sold per agent increased by close to 10%, and that allowed me, to really reallocate 8% of my team working on other support projects that we really need, for our growth and scale projects we're working on for next year. So earlier, I mentioned that MTTR is a really common metric that most people hear. We shaved off six hours of our repair time, and that resulted in an 8% overall improvement in our MTTR, which is our customer impact time. And lastly, you know, we stay focused on our customer experience, and our CSAT rating has been high. It remained high at 87 or 97% right now, and our ticket reopen rate, meaning how what percent of the time do we close a ticket and a customer comes back and has extra questions, and that reopen rate improved by actually 3%. So, overall, the results have been really positive. It allows my team to continue improving internally on our productivity and our efficiencies while still maintaining a laser focus on our customer's experience. So it's a big win for support, and we're happy to continue using it and growing our use of these bots. So thank you. Awesome. Now that those are great results. So, hopefully, as we're winding down, our time together today, you guys are all excited and ready to build your own AI workflows. So, we wanna see, you know, what you do you have any ideas? Do you have, apps that you're thinking about? We would love to understand what you guys are, looking to do. We have internally done not only the support bot. I think, the team mentioned a couple of others. We built a knowledge bot. So this is on our product. We've deployed that not only in our product. So if you are, a Celigo user and you see the knowledge bot, on your, UX, it's all powered by Celigo behind the scenes. And this is gathering all of the information about our like, organization wide. So all of our, help center articles, some internal Slack, messages, the tickets, and the ticket responses. These are all getting, put together and feeding our knowledge bot that then enables our our our own support agents, our customers, and, basically, anybody at Celigo to ask a question about how you do something in the product and get a very comprehensive answer. So that's another, pretty strong use case we've seen in customers that have leveraged the knowledge bot, actually onboard, and get go after new things that they haven't done before at about a 30% faster rate than the folks that are kind of hunting and pecking and doing it the the old fashioned way. The other one is, is more internal. Like, so, all of our customers have account managers. Those account managers, their job is to not only help customers make sure that they're leveraging that all that they can out of the products, but understand what what could be next for them. Like, what what else is going on in their organization that might be a good fit for automation with Celigo? So, you know, before our for account management team, you can imagine your your account management, your account manager, you got a new account. You know nothing about them, and you're basically going into five, six, seven different systems, some of which you don't have access to, such you have to reach out to a support person or a CSM and ask them to get information about the account. And it could take you two, three hours over the course of, you know, three or four days to gather all of the information you need to have a meaningful full conversation with your with your account. And anybody who has account managers or manages customers, has this problem. Right? So it just takes a long time, and and you'll you'll tend to miss information. So what we've bit we we built was a customer three sixty agent that similar to the Slack bot that we just went through, they put in a customer account name into Slack, and it goes off and it gathers all of the information about that customer. So everything in our CRM, everything in the product itself. So who's in the product? Who's building? What are they building? It looks at ZoomInfo information to see what other, technology they may have within their tech stack that that, we could potentially help them with automation and and presents, potential other use cases. So now the rep is armed with not only where that customer is, how what they're doing, you know, any kind of issues that they may have had around support tickets, etcetera. So full diagnostics on, like, where the customer is, the customer health. But now they also are set up to have a a feature facing conversation. And and they are provided, you know, an email, template of, like, how to engage with the customer. And that's actually helped us not not only increase the number of business review meetings that we have with customers, but also drive additional upsell because we're having much more meaningful conversations with those customers and we're providing real value. So that's another bot that we created. So and, you know, the list kinda goes on. So we wanna help you, do some of the stuff that we've done. You know, you, I'm sure, have your own use cases, But we have some templates out in the marketplace. So if you're ready to go, you've got your own LLM or and it could be it doesn't need to be OpenAI. It could be anthropic. It can be Gemini. It can be all any kind of LLM that's out there. Same with with vector databases. We've got templates out here. They're the temp size versions of the some of the flows that you saw today that, Yousef and Sandeep went through, and we're here to help. Right? So we have, we have a number of experts that know how to build these, AI agents and AI workflows that can help you and your team, get going. We'd love to have a conversation with you to understand what you're trying to do and get you going. So please go ahead and visit the marketplace, download that template, and, and reach out to your account manager, and we'll bring, bring experts to help you in. And if you're looking for some inspiration, we've got other use cases. We have, these are a mix of stuff that we're doing as well as stuff that other customers are doing. So there's plenty of examples out there of how people are finding value using AI in their workflows that have been built through, through Celigo. So with that, I that's gonna conclude our time together today. We've got a ton of articles for you. You can see them in the docs section, on the, on the old class cast interface. You can download a bunch of, reports. We have a Gartner report out there that's quite interesting around, AI and once you use agents and once to sort of think about other mechanisms of automation to get the same results. But, anyway, thank you so much for your time today. And, if you, join us for a demo, in September, we'll give you a $50 gift card because we we really wanna hear from you and some of the work that you're doing, especially around these AI use cases. So little incentive for you to, register. Just request just click on that request a demo button, and, we will send you off a gift card. So thank you so much for your time. If there's any questions, I see that there is one here on, oh, there's a few here. So I think you touched on it, and this is maybe back to you, sir. How how we handle the this your Sandeep, as well. How you handle the security aspects. Yep. Yes. We we touched on that one. Do you build this only Zendesk? We touched on this one as well. It's a Slack or any front end that you can, do that you can use. So, yeah, I think, I'm not sure if you have any other questions. These are the two questions that we got. So, yeah, let us know if you are building these use cases or if you need help building this, type of AI agents, we are here to help you and, to support you. Yeah. Thank you for joining. Awesome. Thanks, everyone. Have a great day.