Name: Building a CI/CD Pipeline with GitHub and Celigo
Uploaded: 2025-08-05T16:18:10.097Z
Duration: 50 min 32 s
Description: Building a CI/CD Pipeline with GitHub and Celigo

Transcript for "Building a CI/CD Pipeline with GitHub and Celigo": Hello. Welcome, everyone, to this session, talks tech spotlight. So, we are trying to every month, to have, like, a new topic. Hopefully, this one is an interesting one for you. I see in the chat people from different areas of the the country. I see Nashville. This is the third time I see, a Tennessee, someone from Tennessee. I think Alexia is already from Knoxville. Nashville, I see Nashville as well. Another Knoxville. Great. Good to see people from Tennessee. Okay. Let's wait for maybe, give it a minute. Let's see if we can have more people join in the the session before we can start. Yeah. Munich. Mention. I have a lot of family in, South Of Germany. Great great country, especially Munich area. Yeah. Great. Well, okay. So, I think we, we we can start. I think we can start. So welcome again, to, this session, one of the tech spotlights sessions. This time is a little bit different. We we are going to explore a new topic, the CICD, continuous integration, continuous develop the delivery, pipelines. I think for those who who are, like, familiar with the developments, coding, and DevOps processes, this is, one of the most important pieces of, the processes needed for the developments, especially when we are doing some software. We are we are going to touch on that, in a lot more details. So before we, we get to our topic today, so please, if you are interested in looking at other very interesting topics, so today, we are going to cover the CICD pipelines. Next month, we have the API builder, lookup cache performance. So this is the you want to have APIs in house in your company, organization, this one will help you, this session will help you explore how you can build those APIs in Celigo and integrator.io. And the lookup cache, we we we, we wanted to, to have the lookup cache as part of this API builder session because the lookup cache is very, very important to enhance the performance of those APIs. And it's a good, opportunity for us to show you this new kind of new feature. I think it's, few months now that we, released this. So, you can build the the API and and use the lookup cache to enhance the performance. So and and and reduce the number of calls that you do to different systems. In October, we are going to do the parts two. So the part one will be the build of the API. The part two is the API management. So the API management layer is more of, socializing, that API that you built, things like securing the API, monitoring the APIs. So the it's it's kind of like September and October are both focused on APIs. The first part in September will be the build. The second part in, October will be everything else from money from monetizing, socializing, the the APIs, monitoring the APIs, and in securing those APIs. So, how do you want so for example, you want to build an API, you want to have an auto authentication as an OAuth two point o instead of just a a bare auto authentication. That session, will show you how to secure those APIs. Okay. So, let's start to today's agenda. So we will, start by talking a little bit about DevOps in iPaaS, and why we are going to, why do we need to use the, DevOps in in an iPaaS. We will do some recap on the integration life cycle management. This is the feature, embedded in the Celigo Integrator IO. And then we will see what Celigo offers to, to fulfill the requirements needed to build those pipelines, the CICD pipelines. And then on and then, we will, move to, a reference for the, CICD, continuous delivery pipeline. I will show you an example that we have, we built for GitHub actions using GitHub actions, and using, Celigo flows. Okay. So next, have a poll question. I'd like just to have a sense of how familiar you are with DevOps in in in inside your, your organization. Do you use DevOps processes for your developments? Is it like something required? Do you use it only for specific, projects, or do you handle the developments, internally? I think you have a poll question already. Let's see. I think you have the polls on top right so that you can answer this question. Can you see it? One second. What about now? I think you have it on the top right. Okay. So does your organization already use DevOp processes for your developments? Yes. We do for all the developments, only for specific projects. We don't handle developments internally, so you have some, consultants, or, like, an external, organization doing the developments for you. Or you do the developments, but you never use, DevOps as part of your process. Okay. You see we have, almost, like, 25% or more with, like, people are using the DevOps in some projects or in all the developments. Interesting to see that. So, yeah, means this session is something that will probably help you, in, treating the or handling the the integrations as part of your development projects. Okay. Perfect. So, let's go and let's start. So our goal today is simple. So, we will, we will show you how Celigo's integration lifecycle management, ILM, can feed right into GitHub. So every integration revision or snapshots that you take in your integration life cycle management, in a governance repository, NIPs and then governance repository. And in the next, thirty minutes, we will cover why this matters, what minimum pieces you need, a reference pipeline you can copy, and the the live demo comes, after that. So follow we will follow that by, an open q and a. And, so let's start with the big why. Why should, integrations be treated like code at all? Right? So the the the main thing here is that we need to treat the to handle, the integrations as, as a software asset. Because at the end of the day, it's a software, and we have the possibility have the possibility to extract or export those integrations as, a zip file with, terms of, JSON file. So at the end, it's software that will, will be, handled, in the back end by by Zaligo. So it's not it's not any so they are not, a throw away scripts anymore, and they are mission critical, those integrations. If a flow breaks on, on a Black Friday, for example, you need a proof of exactly what version ran at that moment. Storing snapshots in Git gives you this, immutable history so that you can check that and, see which version broke that, that integration. And this is like, you have a piece. You will have already that revision or that snapshot inside Celigo, but sometimes you need to have this audit trail outside of Celigo, external storage, because your company simply, the security department or organization in inside your company, is requiring you to store those, assets outside of Celigo. So, you can take that piece of software, that snapshot, that that integration, and store it outside in of Celigo for for edit and security, reasons. So, the benefits of external version control, as I mentioned, you have centralized code storage like in GitHub, GitLab. I see a question from Kate already, mentioning Git repo in Azure DevOps. Yes. You can use Azure DevOps. And by the way, we have, a prebuilt connector for Azure DevOps. So you can do that. You will use you would be able to, to have multi environment consistency, because when you, you you you handle that version control external to that platform, you can have more control on what to import and have that human, approval, in the in in that loop that can take the the asset and install it in a different, environment. And you can have a cross team collaboration. So you build in dev environment, and and you want the QA team to be able to to test a specific version so they can just take that from Git and and install it in, the QA environment. And then move everything once it's tested to the the production environment. K. So in quick recap of, ILM. So ILM offers, different options, revision history, revert. You can, revert to different, previous versions. You can do the merge, between different versions, and be able, like in GitHub, see the diff between the different versions that you have of your integration, what changed exactly, and you can take snapshots, which is very, very important. So this diagram is integration life cycle management in one one picture. Each time you are done, with the development, of your integration and to create a snapshot, which is as can see here, number four. So, every revision of that snapshot, can be like a portable asset. Think of a revision as commits and the snapshots as a tagged, releases. So the integration life cycle management user interface also lets you, merge, clone, revert, and it's a git like type of of version control end, but it's inside of the of Citigroup. Yeah. I mean, the we have a question from Maxim. What if we have prod and stage only? So you can you can yeah. You can I mentioned QA, dev, and prod because we we support today the multi environments, multi environments more than just sandbox and production? But if you have only sandbox and production, you can also use that, the same way. Okay. So now let's see let's see which DevOps boxes, Celigo already ticks and where humans are still in the loop, what is still manual in all this process. What Celigo offers already, we have things like, snapshot. The good news is that Celigo already gives you, like, three essential out of the box features. The first one is snapshot export API. So you can, you can use an API. You can export a snapshot of the integration. This is basically the start of today's show. So I will see why we will leverage this very, a lot. The second one is, webhooks, so that Celigo can, can can call, GitHub and, can be called from any other, outside, like, DevOps, a platform. A third one is the documented rest layer for automation. So we have a comprehensive APIs of the platform that can cover, a large parts of, like, we can cover a lot of a lot large of features and components for of the platform, and you can leverage that REST API layer to, to build your automations as scripts, workflows, whether in GitHub actions or any other platform. So everything you see in green, is shipping today. So we have it as as part of the platform. The only piece, that remains gray is the automated input, which is by design and this is deliberate. You can export your integration as an asset to, your external storage platform, but we keep it deliberate to keep the import of that those assets less automated and more manual, and I will explain in a second why. So most customers still want that human approval gate. So you don't want to automate the whole pipeline, the the integration, and also the deployment. So what we do is more we automate the integration of your code into the into the external storage and also the delivery of the code so so that you can create that assets that zip, needed for for to be imported in the platform and just, upload it so that you can use it, in the in the platform as one one of the integration types. So what exactly stays manual, and why is that a feature and not a a bug? This is the part where we so as I mentioned, when once you, store your code as as an asset, like, you store it in GitHub and then you build, that asset, which is the zip code the zip package needed for the, for the deployment. The your operations, whoever is, is handling that. So after the pipeline procedures, the re release the zip, someone from the ops, QA, or release manager that you have in your company clicks upload in Celigo, that upload button, which you can see there. And and that step enforces change control policies. So suppression of duties, some companies, they this is like very this is needed for the SOC two compliance, and you name it. So all these, different compliance, security compliance, aspects, This is very important for, for a lot of, companies to to have this human in the loop. So everything up to the that button is automated, and everything up to that remains under your governance model. So let's zoom into how the automated part actually works end to end. So, this is, a diagram explaining how this is going to work, in the platform. So you see, this is in San Diego. This is one flow. We will retrieve in that flow, this is like a 10,000 foot view. On the left, a San Diego flow, detects a new snapshot. You see that, the the left side and fires a repository dispatch event in GitHub. So GitHub or GitHub actions, has, different workflows that can be triggered using a webhook. So Celigo will take that snapshot and then, from there, will, will be will try to, to trigger that webhook or that, flow in, in, in GitHub actions. The next was the the repo is the the webhook action handles the heavy lifting and takes that workflow. You see in blue, we have two workflows there. The the first one, the first one will be the the CI job, which is the continuous integration job, and then the CD, which is the continuous delivery job. And then with the attack release, assets, so at the end, we will have, like, an asset, which is a package, a zip package that can be stored, in in GitHub and can be reviewed by the different, and have, like, kind of, like, for for audit purposes, have that asset available for whoever wants to audit the integrations. So let's break those two jobs down. And, first, let's start with the continuous integration job. So first thing that we have here, step one. So this is the job. The first one, it will be a a YAML, file or workflow in in GitHub actions. Step one, trigger and check out. As soon as GitHub receives the webhook, we spin up a runner and pull the the repo. Step two, we fetch the stage that, snapshot. We grab the signed URL that we sent. So you see in the bottom of this, the button that JSON, this is what we are going to send from Celigo to the GitHub actions, workflow. So we are sending things like the, the signed URL, which is very important for the the workflow that scripts living in, GitHub actions to be able to download that snapshot. We grab that signed URL, unzip the snapshot. We download it. We unzip the snapshot and we create, snapshot folder in, and create a branch in in GitHub. And then we push a commit to to GitHub. The step three is, the pull request and the validation. So a pull request opens automatically. If the branch, protection rules are are satisfied. So GitHub merged into main, lining us up for a for a delivery job. So, now let's see. Please, if you have any questions, just put them in the q a section, and I I will try to get, into, and respond to all the questions regarding this. I think this is a more more these slides are more the theory part, but we will have a demo where I will more, deep dive into all these workflow scripts and how they look like in GitHub and, what is the flow triggering these, GitHub scripts in in GitHub actions look like. So I will answer all the the questions that we have in q and a, and I will show you in details, during the demo, the details of those, those, workflows. Okay. So now let's let's see how that merged, code turns into, a release artifacts. This is the second part, which is the continuous delivery job. So, in in in this part, so here, we have, like, step first. First step is the trigger and prepare. So every push to main kicks kicks this off. And, the second step, we, we package the snapshot. We zip the the newly merged and snapshot folder and upload it as an artifacts. You see in the bottom here, this is, the artifacts that you would have. It's a zip code the it's a zip package, with all the, the the integration that was downloaded from, from, from Celigo. The third step, we create that release that finally, we tag it with give it, like, some names significant name with the date time. This is how I'm I'm doing that here. And and, and, yeah, we we upload that in, in GitHub. It's it will live in GitHub and can be downloaded from GitHub. That zip is your golden package. So audit ready, rollback, ready and one click away from from input. So you can just basically download that and go to the, steady gold and using that upload upload that package, to be able to use it. So, with that, in the pipeline so let's let's go to the demo. Maybe mix it less theory and more practice. Okay. Okay. So I will start with integrator.io, and I have my GitHub open here. So you will see that I have a GitHub, called Citigo Public GitHub Integration. I have workflows. Those are the workflows that we, we have under actions. So if I click on workflows, you will see that I have two YAML two YAML files. The first one is, the one that we are going to use, the repository dispatch one that we will use to, get that s three signed URL from Celigo and unzip it. We clean it, and then we create, like, a git branch, and then we do a pull request. You see that we are using the GitHub CLI here, pull request, and then we merged, the pull request into the main, the main branch. The second workflow is, the one where we are going to, to create that package to release, needed, in our in our workflow. So everything here is available. I think in one on the right, on your right right of the your, your screen, you will see I think I have here CICD, Celigo, this article explaining everything. And if you look at it, so it's basically almost the same content of this session. So under this, part here, you will see external code version of system. It will take you to the, there's another one. One second. So the GitHub one. I think it's already in, the section. Yeah. This is the one. It will take you to the Celigo's GitHub, official GitHub, the public one, and you can take those examples of the those workflows from there, and use them, to, to basically build your own pipeline. K. So as we, we mentioned during the presentation, we need two things. So this is the integration that we are going to use as an example. I have this one. It's called Amazon s three. I see. So I have some flows here. I have a revision tab. So, what we can do is, let's suppose that I'm doing some developments on on this integration, and I will just change the name of the flow. It's not a a big big thing to change here, but, like, let's suppose that this is the developments. We finished the integration, building the integration. We are happy with what we've built, so I will create a snapshot. So the snapshot will give it just a name. So here, 25, 25. I tested this just this morning, so let's do number two just to have okay. So we have a new new snapshot. So what will happen now is that we have another flow. I have another flow that will get that latest revision from the the point into this flow is pointing to that integration with the ID. Right? I can pull the the revision. I will get the signed URL using the API, Celigo's APIs. And then I will use the trigger, the, GitHub trigger to trigger that, event in my, GitHub. So if I look at the mapping here, what I'm sending is basically I'm sending, the URL. I'm sending the ID of the integration, date creation, these type of things. This is the payload that I'm sending to that webhook in GitHub actions. And in GitHub, I'm just using, dispatches, which is the the endpoints in, in in GitHub that's, basically, triggers these actions. Right? And, okay. So I see that my flow picked up already, the the the integration and started already the first workflow, which is the continuous integration. It's did all these steps. So set up the job, downloaded the the snapshot, got the the pull request, and now it's installing the GitHub CLI to create the pull request and do the merge. And then once this is done, it will go to the second step where we are going to do the continuous, the continuous delivery. Okay. So this is done. The install is done. So auto merge is something is happening here that was not supposed to happen. So let's see what's the error here. Process completed by exit one. Okay. So I think we are using another branch here that should not be there. So let me just remove this branch because I used this earlier this morning. Those are the good things with the live demo. Okay. So Sorry to interrupt you here, but the screen share here is frozen. Some chats were coming in. Just letting you know. Oh, okay. I think I did one. I'm so sorry for that. Again, screen share. Sami will do the whole screen. Yeah. No worries. I'll jump back on. But thank you. Alex Alexia for, reminding me. So let's go back to, to this. So, in, in GitHub, so, we have our workflows that you can use. So in workflows, you can see I have two workflows, YAML, files. The first one is the dispatch repository dispatch where we have all the code needed to pull the file, to pull the, the snapshot URL and, do, all what is needed, to create the pull request and merge it to the main branch. The second one is the build and release. So we build that artifact and we release it as a release in in, in GitHub. So as I mentioned before, this, two, workflows are available for you in Celigo's, Celigo's public, GitHub. It's just here. Okay. So what I will do so I will, again, create another snapshot. Let's do this time it's, and twenty twenty five three. This is my third, snapshot and I will run this flow. This flow will take the revision and will create the sign pull the sign, URL of that revision for that snapshot, and then we'll trigger the dispatch event in GitHub actions. So we should be able to see under actions, this one. So the first attempt was not successful simply because I had another branch that I used this morning, and it was not, it it prevented the the merge, with the the main one. Okay. So let's go back to this. Yeah. Let's see. I think I had some queues, or ID queues. So completed the first one. The revision was completed. We got this fine URL and then and then we have to want to dispatch now the events, in GitHub actions to trigger that, workflow that we have in GitHub actions. Okay. So where the workflow is triggered now. So if I go back here, I should be able to see now continuous integration events run-in. So, in this continue to continue integration, integration, I will be able to see that I pulled all the job, the got the snapshot, got the the s three, URL, got the snapshot, and now it's installing the git, GitHub CLI because we will need it to do the pull request and the auto merge. Why we are doing this step here in our script, it's because GitHub does not support to, doing this using, using the API. So, this step is very important, installing the GitHub CLI and using it in in our script. Hopefully, everything's will go as expected this time. Yeah. Have to wait a few seconds. See that's installing everything here. The package. E dot okay. So now it's creating the pull request, and we have an auto merge again issue. Okay. So I'm not sure what is the problem here. Those are the nice things with the, again, with the live demo. K. So I think I have a pull request here. Maybe I will just, delete. Looks like something is wrong. K. GitHub is not happy. I can't open even the pull request. That's new to me. That's new to me. Anyway, so I have I have an example that I just did this morning. So, it should look something like this. So I hate to say it, but GitHub has some issues. So that's even when I'm trying to open that pull request right now, something is wrong apparently in in GitHub right now. This is something that I just ran literally few minutes before the, this presentation. It should create that per request, create the auto the merge and post that to the main, to to the main, branch. And then the the second workflow is this one. It will create, build that, the the artifacts needed. And, and then, if we go back to this, the second part is to release. So we'll release that as an asset. We'll release that as an asset. Where are we going to find the asset? We go back to the code. You see here the the right part, the right side here. There is a release main. So this was released you see it was was, like, released just forty five minutes ago. We have a deploy dot zip dot zip. This is our asset. So if you want to use it for the, for, your integration, you can just download it like this and take that, that zip and use it in, in Celigo, under here. And you can just upload it as an integration. So, this is how we build this type of this type of integration platform. So there are a couple of things that I want to mention here. This flow is not mandatory. It's optional for you. You can use this flow because I'm using Zilligo, so I want to do the, like, this part as, as Zilligo's, because I have access to the flow so that I can just use Celigo to build my my my kind of my logic. But you can do it in a script if that's that's something that you want to do. So, getting we Celigo offer all the rest APIs needed to, to be able to build this. So you can just build a script that takes the last revision, the last snap snapshot sign sign URL, and build that, JSON so that you can trigger the dispatching, GitHub actions. But in my case, I prefer to build it, in the platform. You see that I have this, every five minutes. So it's pointed to my integration, and every five minutes it's pulling this to get the latest version of the snapshot and trigger this. If nothing if I don't have any, new snapshots in the last five minutes, it will be just ignored and will be, I will not get a it will not trigger anything here. Okay? So let me go back to, the platform here. So, and let's go back to the slides. Some key takeaways here. So, first of all, I I I think that now the using the snapshots in GitHub is something that is that is very recommended right now. I mean, you if you are using if you have, like, these type of audits running on your, organization, you want to have that asset at the end that can track the latest changes happening on that integration. Celica automates everything, up to the deployments. It's by design, so we don't want to automate them, auto deploy. We want to keep the human in approval. And, and then you can, you can if you want to build this, you can replicate this, the this pipeline, you using, the assets that I we have in that, article I mentioned. You have access to the GitHub, Celigo's GitHub, repository and, and you have also the the APIs needed to build that that type of, YAML. With that, I will think I would take some questions. Let me go to the q and a. Good. So we have some questions here. I think I answered the first one, which is the Git repo for Azure DevOps. Yes. That's something that we, we we support. And, let's meet. So what if we have prod introduction? I think we answered this one. Is the ILM applicable for all flows including the prepackaged integration, apps? That's a good question. So, the ILM is covering most of the flows. I think there is in the road map a new, a new enhancement to the ILM to cover also the integration apps. I don't think it's covering every aspect of the integration app because the integration apps has some type of configurations, and those are not handled by the, the integration life life cycle management, the ILM. But there is an enhancement coming very, very soon to cover that as well. Patrick said, this is just another layer of complexity. So if you don't have, as I said, during the the presentation, if you don't have a need for audit, external audit, keeping a track of the changes externally, this is maybe something that you don't need. Not every organization, not every company needs, to have a pipeline to, keep their integrations, in an external storage. But a lot of companies with, this is record requirements. And I had a lot of calls with a lot of our customers and partners where they say clearly that their security organization, is requiring them for every piece of software that they have to have an external storage for audit, for security reasons. So, yeah. I I agree. It's not needed by everyone. And maybe it's can be just add, like, layer of complexity for some some of, the the the customers out there. Could it be a merge conflict? I've been getting merge errors quite often. Maxim, I honestly don't know. I will review that after the call. I I I was running the the workflow many, many times the past, days, and I never had that merged conflict. I I believe it's something related to GitHub right now because you saw that error when I was trying to see the pull request. For some reason, it was saying that the error is not, that that I can something is something wrong happened, on GitHub site. So I'm not sure 100%. Do we have any other questions? Please do not hesitate. I would double check quickly the the chat. Yeah. I think we answered all the questions. I'm still waiting for the multi sandbox feature. So, if you have yeah. Mark, so regarding the multi sandbox feature, it's already there. Please, get in contact with your account manager or your customer success manager, and we will, see if we can help you, get that multi, environment feature to your account. It's related to different other aspects like the licensing. So we can double check on your account and see if, if this is just like something that needs to be, activated on your enabled on your account, or is it something that's, related to the license? I think we covered every question. I I have one from Keith. Is your example for a specific integration or can you have it run it on all of them in your environment? Creating a new integration will automatically add to get yeah. So it's a good question. Let me share my screen again. I like to share my screen whenever it's needed. So entire screen. Okay. So this integration this this example that I have here, I'm pointing to one integration. Right? I'm pointing to this integration. It's like I'm pointing to one integration tile, and I want to have this flow specific for that integration tile. But that's not, I mean, it's you are not it's it's optional for you to do it this way. You can use the Celigo API. Rest API. Let me pull quickly. So in Celigo Rest API, under integrations, you have the possibility to pull all the integrations that you have in your account. You see? Slash integrations. Get all the integrations. So once you get all the integrations so it's basically, it's going to be one step before this one. You get all the integrations. For every integration, you get that ID. And here, instead of using the ID in the relative URI, you will just use something like, record dot integration, ID. And it will it will pick up for all the integrations that you had in the previous step, the ID, and take the revision and create for every one of those revisions, the snapshot URL and trigger for that the, the GitHub actions. So the APIs are there. You can just, based on the logic that you want to build, you can build the your own logic. If you want to point to one specific integration that is the most critical for you, you can just use this one, same as I did. If not, you can you can use slash integrations and pull all the integrations in your account. So, I think we we used, in that example, once you get the integrations, you use this one here. Slash integration slash ID, slash template to pull the revision. Okay? Let me go back and see if you have any other questions. If not, thank you again for joining us. And hopefully, that was, helpful and useful for you. I understand your points, Patrick. This is not for everyone. This is for those companies with specific needs of security and audit. That's the main thing. Because the integration life cycle manager embedded that we have in in in Citibul can cover all the needs, git like type of, operations. But if you need to store it outside in an external storage, this is what you need this type of, CICD pipelines. Okay? If you don't have any questions, thank you again for, joining us. And, hopefully, you can join us for the next one with the API builder and API management. Thank you.