CircleCI + Github actions

Overview

Three days ago, I decided to use CircleCI as a CI/CD tool to deploy my projects. What I was looking for was a way to use Github actions and CircleCI to deploy my projects.

I have a project and have two branches/flavours: production and development. For both of them I want to use Github actions to trigger some tests and code checking whenever I push to the branches.

But whenever I create a release out of the production, I want to do some other stuff, for example in my case I want to deploy my app project to app stores.

Since I was/am completely new to CircleCI, I had a hard time to find a way to do this, so decided to share it in this blog post.

Github Actions

OK first thing first, Let's see how is my working directory:

I have a Flutter app inside app directory and the backend of the app is developed inside the server directory and I decided to have a documentation for it inside docs directory.

So as you can see I cover server and docs of the project in one working directory. But I only want to trigger Github actions for app and not for docs or server. Also I might have different set of tests and workflows for server later and for docs I want to run some commands to deploy the documentation.

First we need to config github actions. Inside Github website, select your project then you can find the Actions tab. Click on it, it might suggest some workflows based on your project's language, and you can search or create one as well.

For me, it suggested Dart and if you press the configure it will create a template for you. All it does is that it creates the files and gives you a template, but we will do it ourself.

Github workflows are stored in the root of your project inside .github/workflows directory and for each workflow you have a .yml file.

I only want to configure github actions for app and want two workflows for two branches. So I'll have two .yml files: app_production.yml and app_development.yml.(You can choose any name for these files)

The app_production.yml file will be triggered when I push to production branch.

The app_development.yml file will be triggered when I push to development branch.

Working directory

My project/root directory inside Github is called sudoku and then inside it I have the 3 discussed directories: app, server and docs.

So I need to tell the Github actions where to look for the files to test and check. Here is my workflow configuration for production branch(app_production.yml):

name: Production workflow
on:
  push:
    branches: [ production ]
    paths:
      - '**/app/**'
jobs:
  test:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: app
    steps:
      - uses: actions/checkout@v3
      - uses: subosito/flutter-action@v2
        with:
          channel: 'stable'
      - run: flutter doctor
      - run: flutter pub get
      - run: dart analyze
      - run: flutter test

This is a very simple configuration, the only important thing about it is how to tell the Github actions where to look for the files.

As you can see I configured the working directory as app

working-directory: app

And will trigger only on push to production, and if and only if we have some changes in app directory, so if I have commits in other directories, it won't trigger.

on:
  push:
    branches: [ production ]
    paths:
      - '**/app/**'

That's it. If you push something to production branch, Github actions will trigger the app_production.yml file and will run the commands and will be successful and you will see the action in the Github UI.

You'll have a Production workflow in Github actions sidebar and a list of actions that ran before. Something like this:

It failed several times! 😄

And if you click on them you can get more information about them.

Let's go and configure CircleCI part.

CircleCI

For CircleCI you need to go to their website and register and then in Github marketplace, set it up for your github account or organization to give it access to the projects.

Finally you need to set up circleCI for the project. Go to CircleCI website and choose projects, and from the projects list, you can choose your project. I logged in with my Github account and I chose Set up Project for sudoku project.

When you click on Set up Project you will see a dialog with the following information:

And as it suggests itself, the fastest way is to have the circleCI config file inside the .circleci/config.yml, so let's create it and after that we will try this again.

I will create a very simple configuration that only runs flutter doctor and echo a message!

Here is the config file:

version: 2.1

workflows:
  production:
    jobs:
      - test:
          filters:
            branches: 
              only: production

executors:
  flutter-exec:
    docker:
      - image: cirrusci/flutter
jobs:
  test:
    working_directory: ./app
    executor: flutter-exec
    steps:
      - checkout
      - run: echo "Hello CircleCI!"
      - run: flutter doctor

Nothing special here, there is a workflow called production and it has a job called test. These names are just examples, you can use any name you want and production has nothing to do with the branch name.

You can check CircleCI documentation for more information. I only want to configure both services(CircleCI and Github action), and connect them together, and not talking about each of them in details. Both services's documentation and tutorials are very good.

OK we created the config.yml file and pushed these files to production and development branches in github. Let's go back to CircleCI and press Setup up project button.

Now you should see something like this:

In branch section, it might show development or main, it depends what is the default branch for your project. But because we only want to run the commands for production branch, we will choose production branch, although in this stage that both branches are the same, and also configurations are not different in branches, it's doesn't matter what branch be selected, but let's choose production branch and press the button.

Now let's review some files that we created, we have config.yml file inside .circleci/ directory. The goal is that it runs some commands on the project, but how should we trigger it?

For github actions, it was inside Github, and it triggers based on the different github action events and we do it on push event on production branch only if app has some changes.

The reason we wanted to use CircleCi was to listen to release event on production branch for app changes.

When you set up the project with CircleCI, a webhook will be added to that project in order to listen to the events and that's how it will be triggered. Also there are a lot of 3rd party services that can be used to trigger the workflow with different cool features. But for Github to check the webhook, go in Github, go to the project, and click on Settings tab of the project and then click on the webhooks in the side bar.

You'll see that there is a webhook from CircleCI.

We will come back here again later, but all good for now.

Now everything is ready. Github actions trigger when we push to branches, and we have a configuration for CircleCI that we want to be triggered when we have a release on production branch.

Let's go to the next section that I wasted a lot of time for it, and it is responsible for triggering CircleCI configs.

Github Actions + CircleCI

Now this is where the story begins. 🥲

Well to be honest, I wasted a lot of time for this section, but I will try to explain it in a few steps.

Let's review:

• We have app_production.yml file inside .github/workflows directory.
• We have a config.yml file inside .circleci/ directory.
• We gave CircleCI access to the project, and we set up the project and have the circleCI webhook added to the project.

To trigger circleCI by Github actions we can use Trigger CircleCI Pipeline. You can read about it here, BUT I believe it needs to be updated, and for me as a beginner it was hard. Even though the linked tutorial has two examples for it, it was still hard for me to figure out where is the problem so that it doesn't work.

When you go to the marketplace you can see a list of all the steps that you need to follow to connect Github actions and CircleCI, but I'll explain it a little bit as well.

First we need to create a new workflow in Github actions, this files is similar to app_production.yml and app_development.yml files. The only difference is that we want to trigger it on release event on production branch. I called it circleci.yml and it is inside .github/workflows/ directory, but you can call it whatever you want.

name: CircleCI
on:
  release:
    types: [released]
    branches:
      - production
      
jobs:
  trigger-circleci:
    runs-on: ubuntu-latest
    steps:
      - name: CircleCI
        id: production-release
        uses: CircleCI-Public/trigger-circleci-pipeline-[email protected]
        env:
          CCI_TOKEN: ${{ secrets.CCI_TOKEN }}

Please read the step 1 and 2 of the marketplace to understand what is the CCI_TOKEN and how to get it and to set id and name for pipeline, but I will explain it a little bit.

This id and name can be whatever you want, but it is important to set them, because it will be used to trigger the pipeline.(We will see this later)

For second step, we need to set the CCI_TOKEN, you should navigate to your project in Github, then select Settings tab, and in sidebar , select secrets and click on actions button and add a new secret called CCI_TOKEN.

Now we need to update circleCI config file to trigger the workflow based on github actions.

Here is the updated version, and I'll explain it:

version: 2.1

parameters:
  GHA_Event:
    type: string
    default: ""
  GHA_Actor:
    type: string
    default: ""
  GHA_Action:
    type: string
    default: ""
  GHA_Meta:
    type: string
    default: ""

workflows:
  production:
    when:
      and:
        - and: [<< pipeline.parameters.GHA_Action >>]
        - equal: ["release", << pipeline.parameters.GHA_Event >>]
    jobs:
      - test:
          filters:
            tags:
              only: /^v.*/
            branches: 
              only: production

executors:
  flutter-exec:
    docker:
      - image: cirrusci/flutter
jobs:
  test:
    working_directory: app
    executor: flutter-exec
    steps:
      - checkout
      - run: echo "Hello CircleCI!"
      - run: flutter doctor

First change

The first part is this new parameters:

parameters:
  GHA_Event:
    type: string
    default: ""
  GHA_Actor:
    type: string
    default: ""
  GHA_Action:
    type: string
    default: ""
  GHA_Meta:
    type: string
    default: ""

If you check step 3, you can see some information about it. Whenever Github action call the CircleCI via webhook, it will send the following information and CircleCI get them as parameters to filter workflows and jobs and etc.

Later I'll show you how to check what information will be sent to CircleCI via Github, but for now I want to explain what are each of these parameters.

GHA_Actor : This is the user that triggered the event. For example, if you push to the production branch, then this is the user that pushed the code.

GHA_Action : This is the action that triggered the event. This is the id of the workflow that triggered the event, the id that you set in .github/workflows/circleci.yml file. We will use this id later to avoid double triggering as mentioned here, but for now it is not important, I'll explain it later.

GHA_Event : This is the event that triggered the CircleCI. For example, if you push to the production branch, then this is push and in our case it is release, because we only will trigger the CircleCI when we have a release on production branch as we set in .github/workflows/circleci.yml file.

GHA_Meta : This is the meta data that is not required for the CircleCI, but you can pass custom attributes as well. Here talks about it.

Sample

For example here is a successful action sample that triggered CircleCI and the parameters that were sent to CircleCI. This is from selecting the circleci workflow in the github actions and then clicking on one of the actions.

Second change

As the second step for changing the CircleCI config file, we changed the workflow to be triggered based on github actions parameters. we use when to run the workflows based on some conditions.

...
  production:
    when:
      and:
        - and: [<< pipeline.parameters.GHA_Action >>]
        - equal: ["release", << pipeline.parameters.GHA_Event >>]

Here I set two conditions, first one only runs this workflow when the GHA_Action is from the webhook, and the second condition check if the event is release.

First condition based on the documentation is used to prevent double triggering. And the second one is useful when you have different events and based on them you want to trigger different workflows. For example let's say we have a push event and a pull_request event, and we want to trigger different workflows when we have a push event, and when we have a pull_request event. But here we only have release event, but I set this condition as well. No harm to set it, but it is not necessary.

The first condition had error in the documentation, and I put some times to understand how should I configure it. There is a PR for them that will update it soon. Here is the PR.

Also if you notice, you see that I add another filter to the job with name test:

jobs:
  - test:
      filters:
        tags:
          only: /^v.*/
        branches: 
          only: production

Well this also consumed a lot of times! I wanted to run this CircleCI job only on release event, so my release has a tag like v1.0.0 and then I go to releases page in github, and create a release from it. I did that a lot but I faced to this in the CircleCI website. No workflow.

This means that none of the workflows will be triggered, because it doesn't match the conditions, so it is ignored. And it should be the case, because we want to trigger the workflow only when the workflow conditions are met.

But what is the problem for our case? Well check this tutorial. As you can see when we have tag we should filter jobs and workflows because:

CircleCI does not run workflows for tags unless you explicitly specify tag filters. Additionally, if a job requires any other jobs (directly or indirectly), you must specify tag filters for those jobs.

I changed that as well but still was facing to the No workflow error! In the end I understood that I should add release for the project in CircleCI webhook setting. So go to Github and in the webhook settings, press edit and check the release event as well.

This was the default chosen ones for me:

Now if you push something to the production branch and creating a tag like v1.0.0 and then creating a release from it, CircleCI will trigger the workflow. AND you'll get your successful build.

You might get it on the 333th attempt and on the 74th release!!!!, but for sure will get it in the end. 😂

Webhook API Delivery

For CircleCI's webhook API you can check the CircleCI webhook API documentation.

But for checking what is sent and what is the response via Github, you can navigate to project's Settings and then click on the Webhooks and click on the edit for CircleCI one, and choose the Recent Deliveries tab.

Conclusion

With Github action alone, we can do a lot more, but I just wanted to learn about CircleCI and it is a great tool for CI/CD.

If you found any mistake or have any suggestion, please feel free to contact me.

Happy coding! 👩‍💻