Name: Striking the Right Balance: Simplifying Integration Flows Without Compromising Functionality
Uploaded: 2025-03-04T16:16:09.665Z
Duration: 49 min 44 s
Description: Striking the Right Balance: Simplifying Integration Flows Without Compromising Functionality

Transcript for "Striking the Right Balance: Simplifying Integration Flows Without Compromising Functionality": Good morning, and welcome to the second session of our spotlight series. We are excited to have you here. Last month, we kicked off this series, and it has been great to have everyone you again joining us today. Today, we have another exciting topic, which is striking the right balance by optimizing and simplifying flows without compromising any of the functionality. So let's dive in. So here is the agenda for today. First, we will start with brief introduction, then we will, followed by a deep dive into today's topic. We also have some live example for you. And after that, we will wrap up the session with interactive q and a. So moving to the next slide. Okay. Hello, everyone. My name is Kevin Shah, and I'm a senior solutions consultant with Zelligo. And today, I have my colleague, Yonton, who is a senior developer. Hey, Yonton. Do you want to quickly say hi to everyone? Hi, everyone. Anton Miller. I'm a senior developer here with the professional services team, and I'm based out of Asheville, North Carolina. And, I think we I think we should have a lot of, partners on the call. So before I joined Zaliga, I was an implementation partner. And I kept doing projects with Integrator, and I like the product, so I started working here a few years ago. Awesome. Thank you, Anton. So here is the overview of our future session. So looking ahead in our next session, we will explore how to empower your operations team to take the lead in managing integration, driving greater business efficiency. Then in May session, we will dive into smarter ways to approach EDI that gives you more flexibility when creating partner integrations. And finally, then in June, we are June, we will dive into handling complex foreign validation. And again, we want this program to be dynamic. So we encourage you to provide feedback. And based on your feedback, we'll finalize our topics after June to ensure that we are covering what matters most to our community. So moving so let's do one thing. Before diving into today's topic, let us have one quick poll. So the poll question is, what is the biggest challenge you face when managing integration flows? And the five options are, one, maintaining performance and speed, reducing technical depth, simplifying the flow design, troubleshooting errors, and scaling for, larger dataset. So, yep, let us give thirty seconds. The poll is open for you to select one option. Awesome. Awesome. So most of the people are seeing reducing technical debt. Okay. Awesome. So so the good thing is that today, we are going to cover most of it moving forward into our presentation. So moving forward, let me go to my next slide. So now let's talk about why integration flow are so crucial for modern businesses. As organization continue to scale, they rely heavily on applications and system. For example, it can be CRM. It can be ERP. It can be HR HR system. And each of these application do need to communicate with other to ensure seamless data transfer between the organization. And this is where integration comes into picture. The the importance of integration definitely cannot be overstated. Like, without effective integration, companies would struggle with with synchronizing their system, leading to errors, even they may lose business opportunities. And with what integration will do, it will enable them to, reduce the manual work, then, to have data move between the different application in real time. So that so integration is very crucial for each and every business. However, as the number of application grows, it also introduces new challenges. And as we start introducing more and more application with needs to be integrated, definitely, it will, it will make the it will make our, integration architecture way complex. Though something like point to point integration started with simple to integrating simple to application, but soon when we start introducing more and more application, it will become lot complex. And this complexity leads to a problem. Like, as your integrations become more over engineered, there are they are very harder to manage and maintain. You start seeing technical that Things take longer to troubleshoot and so on. Today, we are going to explore how to design integration flow that are scalable and maintainable. Moving to the next slide, let us let us, cover few of the technical, tech let after covering few of the few of the importance of integration, now let us see few of the science for over engineering integration flow. So one thing is visual cutter. As we start introducing too many steps or too many branches into our flow, definitely, the flow become more complex, and sometime it is not hard to troubleshoot it. Second is the difficulty in debugging. When troubleshooting become challenging, when it is more time consuming, it is it is very clear that, that maintaining maintaining this integration is time consuming, and it might have costly downtimes. That is inflexibility. Then modification or if you want to add certain step in your flow, and if it is time consuming or if you feel that, okay, it is altogether another undertaking, again, it is a sign of over engineered flow, which is not so good. Fourth one is performance bottleneck. Like like, whenever you start introducing lot of things in your flow and you do not focus on the functionality, definitely, your your and you your flow will be quite slow. So even that is not good. Eventually, it might start crashing, so which is not good. And finally, high maintenance cost. So see, if you if you are spending too much time in maintaining a flow, that is also not good. Where fixes take longer than required time frame, it is very frustrating for each and every one, so which which is not good. So now as we have discussed the importance of, integration as well as we have as we have discussed you of the science of over engineered integration flow. Now it is time to see see the what are the best practices for efficient integration. And to cover this piece, I would like to call call my colleague Yonton. So Yonton, the floor is yours now. Alright. Hi, everyone. So, let me go to the next slide here. And I wanna go over four four topics on ways to improve, efficiency and and really improve the ease of maintaining flows in the future. So the I would say that's the the common, common factor for these four things that I wanna cover is really as we build flows or if we have an opportunity to revisit a flow that's been built and working for a while and considering if it could be improved, think of okay. What what can we do to make this flow easier to maintain, and troubleshoot in the future? So that that's kind of the the common core of these four four items. The first one is possibly self explanatory, but it it's a really important topic. I would say it's one of the top things that we see when we, when our professional services team is asked to help somebody troubleshoot a flow that somebody else built. This is a a common thing we see, and it's a it's a really important one to resolve, and hopefully this designed properly from the beginning. But if you encounter a flow that maybe wasn't designed this way, this is a a a good first thing to think about. So the topic is using filters, to optimize the data export. And, really, the the focus here is may let's make sure that we're building a flow in a way that we only export as many records from the source system as we need to process in the flow. So any ability to add filters in the, in the export request, whether sometimes we can use a a get request and we can add filters as query parameters, or in the in you query parameters in the URL. Sometimes we are actually using a query, a SQL type query, to export that out of a system. A lot of our customers use NetSuite, and in NetSuite, we can add filters in the, saved search. So really think about, okay, how many filters can we add at the very first step, to avoid processing data that we've already processed and hasn't changed in a meaningful way that we need to process it again, and to make sure that we are, not you know, we we go into environment sometimes where customers have a flow that might process 3,000 records every 15, but of those 3,000 records, only 10 records are new records each time. And they just haven't realized that they could add a better filtering at the at the beginning. So, let's let's take a look. I'm gonna show you I'm gonna share my screen. I'm gonna stop sharing the slide here and share my screen. And we can take a look at a sample flow here. Okay. So here's a sample flow I set up for this call. And we'll we'll talk about a few things. I'll I'll I'll keep using this flow as our example for, for the other topics as well. But let's imagine we have a a multistep flow. It starts with an export step from we export data from some system, and then we take multiple steps to process the data. What we sometimes see is, somebody would have not very good filters or not as good of filters as they should at the export step, And then they'll have many input filters at each subsequent step. So they might say, oh, well, we only wanna process if the customer email is not not equal, not equals test customer. And they would add these input filters on every step of the way. So let's say they export a thousand records, and each of the thousand records will go into each of these steps, and then, the system will review the record, decide it needs to be, ignored, and then the thousand records all go here and that they all go through the process and, you know, 90 997 of them get ignored and again and again and again. And you end up with a really long process that does very little comp the its output is very small compared to the amount of work it's doing and and bandwidth that it's taking. So whenever we can, the earlier in the process that we can filter out, the better. And sometimes we just we can't quite filter everything. For example, this this sample rec the sample export is from a GraphQL endpoint. Specifically, it's a it's a Shopify GraphQL, and I don't know how to filter maybe there's a way, actually, but I don't know I don't know how to filter a GraphQL, query based on the content of the tags. So somebody might say, well, if it has this tag and this tag, then we need to ignore it. In that case, you might be you might be best served by adding a, some scripted logic to parse those tags and compare them based on your rule, as a in a presave script. So that's an option. Another option is maybe it's just a single tag, and you could add a user interface filter here and say, oh, if the, I keep talking about tags. I don't even think I'm exporting tags in this particular one. But let's say, we might say, if the email is something, then we would ignore it. So we could add a filter here. And what also might happen is we might say, we're getting the orders from here, then we need to look up the customer in another system, and based on some flag on the customer record, then we wanna ignore it. Again, we can filter those records out earlier in the process. In this case, we could, actually use oh, wait. I I want I'm I'm, jumping to the next topic. So I'll, I will stop sharing my screen, and we'll go back to the slides because I wanna I wanna introduce the next topic. So the the next topic is related. It's still on filtering data, early in the process. But sometimes, we need to filter the data, not at the very first step. So, again, on the on the first step, we talked about, filtering as much as we can, at the very beginning of the of the data export. And our next topic is, okay, sometimes we need to filter data not at the initial export step, so I wanted to show you a trick for doing that. So let's I'm gonna stop sharing the slides again and share my screen again. Alright. I'm two for two for sharing the correct monitor. Very proud of myself for that. Alright. So let's imagine a scenario where we, we fetch data from one system. We need to look up data in an additional system. And after that lookup, we would want to filter out some records based on some flag. Oh, an efficient way to do that is by using branching. So here I could add a branch. And right now, for my rule, I'm I'm gonna use my rule is I don't wanna run any records if the customer is this test customer 001. So I'll call my branch. Is this test customer? And my first branch is really, my the main branch that I wanna process is scenarios where, it's not the customer. So I say not equals to this email address, and I'll say new customer or maybe not test customer. How about we call them real customer? And then I could have a second branch that is oh, not phone, but email. That where the email does equal this, and I can call that branch, test customer. So now I have two branches, and the data will go you can see, is this a real customer? If it is a real customer, it goes to the real customer branch. If it's a test customer, it'll go through the test customer branch. Now in some cases, I might want the test customer to just skip some of the steps and then rejoin our flow later, in which case I would drag this and I would connect it to the flow later. But in this case, really, what I want is if it's this test customer, I just never want it to get processed. So I have two options. I can leave what I would call a kind of a dead branch. It's a branch that does nothing. The other option that you can use is actually remove this other branch. So records will flow through the first matching branch. So they'll any records where the email does not equal this. I'm gonna save my changes here just so you can see what it looks like. Any records where, it's a real customer, they'll go through here. And any records that are the test customer, they'll just kind of fizzle out right here. They they won't proceed because there is no branch for them to follow. So that's a it's a really efficient way to filter out records later in the process. Okay. I'm gonna stop sharing my screen. Y'all think I can do three three times of sharing the correct screen? Alright. So back to the slides. So the first slide we covered is just the concept of applying filters to the export step to narrow down data that's the the most important. The second slide we covered is discarding unnecessary data later in the process by using branching. And the next slide that I wanna cover is talking about breaking down a multiprocess flow into several flows or or several steps. And this is really a design conversation. There and there's a lot of there's a lot of room for user discretion or or developer discretion here of what what makes sense for for this particular business and a particular business process. I think the the topic of this, webinar is striking the right balance, and and here it's really kind of the you find the right balance for for a particular business and a particular, use case. A lot of times, as here at Zaligo in professional services and, any of you that are, partners that that work with customers, you probably see this. We get to work with with teams at a business that are they are the business process experts. They really understand what happens in their business as some process is occurring. And in they might think of it in their mind as, okay. In in my business, customer places an order. I need the order to go into my ERP, and I need the fulfillment to go to the three p l so the three p l can ship it or the warehouse can ship it. And in their mind, it's a pretty concise business process. And here, I'll switch again to sharing my screen. So here we can we can look at this at a flow. I'll show you the the flow in more detail. So this is a scenario. It's a, a simple scenario, but we we it's I think it demonstrates the multiprocess nature that we encounter sometimes. So here, we would fetch, transactions from an ecommerce platform. The transactions, because they're sales orders, they require a customer. So here we would create or update a customer record in the ERP. Then we would, create the sales order in the ERP. This is, the this example uses NetSuite, and, the NetSuite ERP has this concept of customer deposit. So if we received money from the customer, haven't shipped goods yet, we record the money we received as a customer deposit. Then we create an item fulfillment. That's what the warehouse will use to ship the goods. Then we send the item fulfillment to the three p l, or to the to the warehouse or third party warehouse. And then we might write back to the to the item fulfillment record to indicate that it was successfully sent to the three p l. So we have this flow, seven steps. And to the business user, they might say, yes. That this is one process. That that's one one thing that happens. But to the developer, we might say, okay. We have seven different interactions with API systems, that happen, and they might not all be directly dependent on each other. I would say as a rule of thumb, in most cases, if you see a very distinct business a a very distinct, technical process that could be separated from the rest within a flow, I would consider making that a separate flow. For example, here we can say, okay. Getting the order from the, Etail from the ecommerce platform, Creating the customer, creating the order, and creating the deposit, those all really need to work together. And these three imports each use data from the initial export step. So this the first four steps of this flow, they really make sense together. And then creating an item fulfillment record, that really happens inside the ERP, and it's it's created from this sales order. All the data we need is this in the sales order. So we could say, alright. Creating the item fulfillment, sending the item fulfillment to the three p l, and marking the item fulfillment as sent to three p l, that's really its own separate process. So if I were designing this flow, it'd most likely separate these three steps into a separate flow. So this is a a concise process. It's from the, ecommerce platform to the ERP, and this is between the ERP and the three p l. So that's how I divide that. We see this is an, a seven step process. We have customers that we hop into their environment, and they have 20 step processes. Or we'll work on a scope with them, and they will see us a process with 20 steps, and they'll and they'll think that's totally fine. Look at that. That's to them one business process. But that's where the the slide that Kevin shared earlier about processes that get really cluttered. There's so many steps. It's hard to read through. And it becomes really hard to do testing, and it becomes really hard to do troubleshooting. You might change some field name to play nicer with this step and not realizing that you're breaking something 10 steps downstream. So whenever we can, reduce the number of steps in a single flow, it's good practice. Okay. I'll stop sharing and hop back into our slides here. Okay. So, yeah, this is, breaking down multiprocess flows into separate steps or separate flows. And, really, again, the the ability to better troubleshoot and better maintain them downstream is is one of the biggest benefits here. Think of a situation if for those of you on the call who are partners, think of situations where you're building the flow and but you may not be around to maintain the flow for your customer. So somebody in your customer's business team will need to be able to look at the flow and maintain it in the future. Those of you who are customers, imagine that you're building these flows, but downstream, you hope that somebody at a different business team will maintain these flows, or you might get promoted, or you might go work somewhere else one day. Or, Nick from our team said a thing that I really appreciate. He said, you might build this flow and need to look at the flow again and maintain it eight months from now and not remember why you built it that way. So the if you can break your flows into concise chunks, they become easier easier to test, easier to maintain, easier to troubleshoot. So that is good good practice to keep in mind. And the the next slide is very much along this the same lines of thought. It's, writing clean and maintainable scripts. So as we, sometimes it flows, we'll need to have some, scripted logic, and it's it's tempting a lot for especially for newer newer builders and newer developers. It's really tempting to have a script that works, and it it does the thing you needed to do that you couldn't do without a script. And you're so happy when it's finally working, and you would just wanna move on from there. But I would really stress the importance of write write that script, that future you, eight months from now, like Nick said, or that, another resource if they needed to look at it, even if they might not be technical enough to write a script themselves, but maybe they just need to review it and understand what was going on. Do your best to leave leave your flow structure and leave your scripts in such a way that, somebody else can understand them, and read through them, and, hopefully, maintain them as well and make changes as needed. So I wanna show a few samples of that as well. I'm gonna stop sharing the slides. I'm gonna share my screen. I'm gonna go to the same flow that we've been talking about all day. Alright. So I just had to make up some excuse to write a script here just to have one for the webinar. So let's say we have some data. We want to clean up the data structure a little bit. For example, we, if the we don't love the structure of the data or we want to have some, scripted logic for some reason, We can write, write a script, presave script to to take the actions that we need and return a cleaner version of the data for our for our needs. I'll show you, I'll show you a version of the of a script that I encounter a lot when I'm and I apologize. I don't know who I don't know who all the people on the call are. I don't know what your experience with JavaScript is. So if this is the first time you're seeing JavaScript, I I apologize if this is looks a little weird. But maybe you can still, learn some things that would be helpful. So a lot of times, we need to loop through sections of data, And this is kind of the JavaScript one zero one. Like, the very first time you look at how do I write a loop in JavaScript, this is probably the example you get is you define an index and you call your index I, and you keep your index shorter than you the length of your array. And each time you add, one to your index, this this totally works, and you can iterate through your data and make your changes. And this this script that I have here totally works. But you can see, as we go through it, we're like, okay. I is something, then we loop through again, and then, what do we call our next index? And because we're already using I up there. So then we use j, and then we might need yet another loop further down, and maybe we call that one k. And then each variable or each field becomes this, like, really long string that you're like, okay. It's this part of this loop and this part of this loop and what's I and what's k again. And it can start becoming really difficult even in a like, this is like a 20 line piece of code. And in my opinion, it's barely readable. It was actually painful to write it this way. So, just because this work, doesn't mean you should leave it like this. I would I would, urge you to think, okay. How how can I make this so it'd be easier for future me or for a future resource to look at the script and make a little bit more sense of it? So the the first thing is I would recommend and when you're looping through stuff or whenever you're defining, variables, let's try to use as as much as we can. We'll try to use variable names that are descriptive. For example, order header would be the header section of the order record. Order ID would be the ID of the order. And if we iterate through line items, we might call each iteration line item so it's easy to see what we're looking at. So that would I would say that at the very least, if I can leave folks with anything about JavaScript, use descriptive variable names. And if you wanna go even further than here at the bottom, I have an example. We can even add notations. So starting to add note you can also the topic here is striking the right balance. I would say it is possible to write too many notes. And all of a sudden, your your 20 lines of code turn into a hundred lines of code because you have a novel between all the lines. So there is a balance to strike. But to the best of your judgment, try to leave the leave your, JavaScript hooks in such a way that if you need to look at them in the future, it's easy for you to read some notes. And and just from the notes and the names of the variables, you can make sense of what's going on here. So that is, again, the with the always with the goal of kinda thinking of, okay, who needs to maintain. It doesn't just need to work now and and be able to complete user acceptance testing, but it needs to be maintainable in the future. Okay. I'm a stop sharing my screen and, you know, go back. So that's the slide for, writing a clean and maintainable script. And, really, the the biggest benefit is that it's easier for other people to understand what's happening, and it's easier to troubleshoot troubleshoot and maintain those scripts in the future. Well, live demo. I think that's what I just did. Okay. And now we have a new a new another poll. I don't know. Kevin, help me out. I don't know when to when the poll is ready for folks to, to chime in on, but the okay. Yep. Yeah. Right. Well, as how often do you revisit and optimize your your integration flows? Do you look at flows regularly and think, okay. How can I improve this? Occasionally, just when specific performance issues arise, rarely, only if some major things are changing in your operations, or never. Oh, sorry. I did not mean to click on the next one yet. Alright. And looks like, rarely only for major updates is a common one. And I think I think that's that's fair, really, when you don't wanna you don't wanna change things too often if you have a flow that's already working. But if you know there's a business business change or a major update in one of the systems, that's a a good opportunity to review a flow and consider if it could be done more efficiently be because of this change, or maybe it's just a good excuse to see that you can add some better notes inside your script, or it's a good excuse to see you know what? This one flow can be split into two flows, and it'll each one will work faster and be easier to maintain later. So that's a a fair approach. And alright. Tips and tricks and best practices. So we we talked about simplifying integration flows to reduce reduce complexity, make them easier to maintain, using filters, at the export step, and using filters as branching later, later in the flow if needed. We talked about breaking down multiprocess flows into smaller, more manageable flows, and we talked about writing scripts that are clean with well named variables and easier to maintain. So, yeah, I think here, I'm gonna hand it back to Kevin for the q and a. Sure. Thank you, Yonten. We do have couple of q and a questions, and, let us start answering them. So first one is how can you identify when your integration flow is becoming too complex and need to be simplified? This is a great question. And like I mentioned on one of my slide that we need to look for the sign of over engineered flow. For example, if you have multiple steps or branches, maybe that is the reason why troubleshooting something, is becoming way too complex. Or may because of that, the it is impacting your performance, and there is some performance issue within with with the flow. Maybe you need to I I don't know if you you are following the multistep path, you might you might have to simplify your flow. So the then one thing which is important is, like, maybe, maybe if, even maintaining the flow or if you want if you want to make certain modification, even if that is a headache, definitely, that is one of the sign that tells you that, okay, this flow is complex, and you need to simplify it. So, yeah, those are the points which you need to look look to make sure whether the flow is complex or not. But it is a great question. We do have another question as well. And maybe, Yonton, do you want to take this particular quest question? The question is, how do you resolve record dependency issue in different flow? For example, invoices are created in flow one, and customer payments are created in flow two. How do we ensure that the information arrives in order in different flows? That's a that's a good question, and it it would it's a little bit system dependent. In a a lot of ERPs, you can let's say you have flow one creates the invoices, and flow two will will pick up the invoices that are unpaid and process the payment on them. So and a lot of systems will have the ability to write an export step that, gets fetches all invoice records, which meet some criteria and are unpaid. So flow one will create the records, and flow two will only have those records to process after they're created in flow one. So that's that's one way to do it. If, I'm trying to think of a more complex scenario where it's possible that you you need another piece of information from a third system. I I'm I'm having a hard time thinking of a a more complex one. But, yeah, in in in in a lot of, in a lot of cases, yeah, being able to export the invoices that need to be processed in their own in its own flow, that's common practice. If anybody has follow-up questions, we we can, also follow-up by email later. Yep. And thank you, Yonton. We do have a a couple more question. So maybe, question for you. Before branching the record data, would you use a front end SQL query to either select the data subset then process or provide any data transformation? What's best practice from your experience? SQL using SQL definitely depends on on your export step. But if you have an export if you're exporting from a system that allows a SQL type query, like, from a a SQL database or Salesforce has, Soquel or, I guess, even even GraphQL in a way is, queryable. So whenever we can, really explicitly filter our data at the export step with a query, that's always best practice. The the the more the more we can filter earlier on in the process, the better. Sometimes it we can't fully, filter everything we want from that export step or from the query, and then we may either need to fetch data from another source, to filter by, or we would need to add a user interface filter in the flow. Or, I I would say this is worst case scenario, but but it does happen a lot that we need to add a a scripted, some scripted process to apply the logic for filtering. And then that script could, yeah, we can add a flag that then uses the branching. That would be that's a common, common scenario. Thank you, Yordan. There is one more question which I would like to answer. So the question is, how, one second. Yep. How how do you balance between optimizing the flow for efficiency and maintaining the necessary functionality for your business operation, which is which is a very good question. So I would say the key here is to prioritize is your core business requirement while ensuring that the integration flow remains as simple as possible. Maybe to achieve this, start by clearly identifying the essential functions which are required for your integration and do and ensure that those are maintained without unnecessary complexity. Maybe then you can focus on optimizing the part of the flow that are most critical to performance such as, like, Yonten mentioned, like data filtering, eliminating redundant steps while preserving the feature that are direct least supports business needs. So that is how I would go with this, to make sure that, the fluid, we are maintaining the efficiency, and we are not compromising any functionality. Yep. Thank you. Let me see if we have I don't believe we have any further question. So one thing is that in a in a doc section, we have linked to to our help, Saligo help document, which will help you to fine tune, which talks about how you can fine tune your flow. So please refer to the document which we have shared via doc set section. And in case if you have any follow-up question, feel free to reach out to me, Yonton, anyone. I'm more than happy to answer all the questions. And thank you for joining today. Have a wonderful rest of your day.