Open In Colab   Open in Kaggle

Tutorial 2: Github Workflow#

Week 1, Day 1: Instructions

By Neuromatch

Content creators: Konstantine Tsafatinos

Content reviewers: Konstantine Tsafatinos

Production editors: Konstantine Tsafatinos


Acknowledgments: [Ella Batty, Spiros Chavlis and neuromatch]

Tutorial Objectives#

Estimated timing of tutorial: [15 mins]

In this tutorial, you will learn the workflow for adding content and updating the repository with your new content.

You will:

  • enable github actions for your repository for the new course

  • learn how to create branches and pull requests

  • learn our folder structure and the strict naming convention for the files

  • learn how to update the materials.yml to reflect the newly created notebooks

  • learn the basic workflows

It is recommended that you read through all of this tutorial before starting to make changes, as all the topics are related to each other for a successful build of the notebooks.

Setup#

In this section, we have:

  1. Github Actions: Enable github actions to allow for triggered workflows.

  2. Branches and PRs: Create local branches, push to github, and create a pull request.

  3. Folder Structure and Naming Convention: Guidelines to follow so that the processing and build steps work correctly.

  4. Materials.yml: Add days and tutorials to the materials.yml file.

  5. Workflows: Trigger the workflows.

Section 1: How to enable github action#

Steps:

  • click on the settings tab in your newly created repo

  • click on the actions dropdown -> general

  • click on allow all action and reusable workflows

  • click save

github actions gif

Section 2: How to create a PR#

When creating content you want to add to the course book, you will need to have python and git installed. Afterwards, you will need to create a local python environment with all requirements for your code installed; including jyputer. Before starting to add content, you will need to create a new branch. Instructions here.

Once you have created your working branch, you can then start to work on your notebook. Before saving your notebook and committing the changes to your branch, restart the kernel and clear all cells. Then you can save it, add the changes, commit them, and push to github.

NOTE: It is vital that you restart the kernel and clear all outputs. The processing scripts will throw an error if you do not, and the workflow will not complete successfully.

Be Careful: Remeber to pull the lastest changes from github before you commit your work! This is important, a successful workflow execution will create new commits, and if you do not pull the those changes before your commit your local ones, there will be a commit history conflict; i.e you will not be able to push your changes.

How to restart kernel and clear all outputs:

restart kernel gif

Section 2.1: Creating a PR#

Once you have pushed the work on your new branch, you can create a PR onto the main branch. NOTE: This will trigger the processing workflow for the course content.

Create a pull request:

pull request gif

Process Notebook workflow:

process notebook gif

The process notebook workflow will automatically get triggered once a pull request is made to the main branch. It will create the student, instructor and solutions directories, verify the exercises, and prepare the content for the publish workflow. You do not need to create or touch the aforementioned folders. All your work should be done on the jupyter notebooks. Remember to restart the kernel and clear your output, if you do not, the process notebooks workflow will throw an error.

Section 3: Folder Structure and Naming Conventions#

In order for the workflows and scripts to work correctly, and for your book to properly build, you will need to strictly follow our folder heirarchy and naming conventions. This is what the scripts are expecting, and you will not be able to successfully build your course book if you deviate from the conventions.

  • book/ (do not touch, this get automatically created/population in the publish workflow)

  • projects/ (explained in tutorial 5)

  • tutorials/

    • W#D#_DayName/ -> (ie. W1D1_InstructionsTemplate; This is strict! You must have the W#D#_ prefix followed by the name of the content for that day. The name cannot include special characters (i.e -, _) and must follow the format shown.

      • static/

      • README.md -> (This needs to match exactly!)

      • W#D#_DaySummary.ipynb -> (The name needs to match exactly!)

      • W#D#_Intro.ipynb -> (The name needs to match exactly!)

      • W#D#_Outro.ipynb -> (The name needs to match exactly!)

      • W#D#_Tutorial#.ipynb -> (The name needs to match exactly! You can have multiple tutorials for a day as long as you number them sequentially and add the number to the materials.yml file.)

      • further_reading.md -> (The name needs to match exactly!)

    • static/

    • materials.yml

See a working example below:

folder structure image

To reiterate, you do not need to worry about the student, instructor, or solutions directories. Those are automatically taken care of.

NOTE: W1D2 contains templates for tutorial creation. Where templates were not included, we provided examples from previous courses.

Section 3.1: Cell Tags (markings at the top of the cell)#

The workflows and processing scripts actively make use of predefined tags to determine how to process the cell. There are tags for video embeddeing, slide embedding, titles, exercises and solutions, etc. It is very important these instructions are followed exactly! Any deviations will break the pipeline. More details about the tags and how to use them can be found in W1D2_Tutorial1.ipynb of the template notebook.

NOTE: Please read the details found in the template very carefully. Specifically the section on exercises and solutions. You need to be precise. We have a script to verify exercises that will fail if you deviate from the instructions. For example, the comments in the cells for the exercises and solutions grouping need to be EXACT. See below:

Exercise:

def func(ex):

  # comments describing the exercises

  x = …

  return x

Solution:

def func(ex):

  # comment describing the exercises

  x = …

  return x

This will fail because “comments” is singular in the comments of the solution cell. These checks are in place to ensure the exercise and corresponding solution cells are appropriately matched and coupled. This comment rule applies to all comments in the exercise/solution cells.

The above is not an exhaustive list of the cell tags, it is simply a highlight of something that can go wrong that is easy to miss. Please find the details of all tags here.

Section 3.2: Add Images#

Your images should live in the static directories. They can be grouped into the a static directory per day, or can be inside the general static directory for all tutorials. The important thing here is that you correctly link to the proper URL. You can directly add the following html into your markdown cells in order to add images.

<p align=’center’><img src=’https://github.com/your_org_name/your_repo_name/blob/main/tutorials/static/your_file_name.ext?raw=True’ alt=’image description’/></p>

Ex:

<p align=’center’><img src=’https://github.com/neuromatch/course-content-template/blob/main/tutorials/static/folder-structure.png?raw=True’ alt=’folder structure image’/></p>

We recommend using .png or .gif file extensions.

NOTE: This will not show up in your local development environment until the images are added to the main branch on github. As you can see from the above URL, it is pointing to the main branch of a github repository. If you would like to test it locally, you can use this:

<p align=’center’><img src=’../static/your_file_name.png’ alt=’image desctiption’/></p>

Remember to change the source back to the github url before pushing your changes!

Section 4: Add Content to the materials.yml File#

The materials.yml file is used in our scripts to help verify links, populate the books, and create our table of contents. Be sure to precisely follow the structure outlined in the file. If you look at the materials.yml file for this template notebook, you will see the structure laid out in order for this book to properly build. There are comments in the file to further explain all the fields you will need to use. Here is an overview of the structure:

  • day: W1D1

    • category: Course Content Template Instructions

    • intro: https://www.youtube.com/watch?v=KxldhMR5PxA

    • intro_bilibili: https://www.bilibili.com/video/BV1HT4y1E7U4/

    • name: Instructions

    • outro: https://www.youtube.com/watch?v=KZQXfQL1SH4

    • outro_bilibili: https://www.bilibili.com/video/BV1vv411i7SG/

    • playlist: https://www.youtube.com/playlist?list=PLkBQOLLbi18ObAiSOZ42YBwOQIKNvspeI

    • qa: #insert as many as Q&A video links as needed

      • Insert Q&A video link 1 here

    • slides: #insert as many as slides links and titles as needed

      • link: https://mfr.ca-1.osf.io/render?url=https://osf.io/rbx2a/?direct%26mode=render%26action=download%26mode=render

    • title: Intro

      • link: https://mfr.ca-1.osf.io/render?url=https://osf.io/6dxwe/?direct%26mode=render%26action=download%26mode=render

    • title: Tutorials

      • link: https://mfr.ca-1.osf.io/render?url=https://osf.io/jdumz/?direct%26mode=render%26action=download%26mode=render

    • title: DaySummary

      • link: https://mfr.ca-1.osf.io/render?url=https://osf.io/9hkg2/?direct%26mode=render%26action=download%26mode=render

    • title: Outro

      • link: https://mfr.ca-1.osf.io/render?url=https://osf.io/2esh5/?direct%26mode=render%26action=download%26mode=render

    • title: Reading

      • link: https://mfr.ca-1.osf.io/render?url=https://osf.io/5vj73/?direct%26mode=render%26action=download%26mode=render

    • title: TA

    • tutorials: 1

Remember to create one grouping of the above content per day and be sure to set the number of tutorials.

Section 5: Workflows#

There are two main workflows used to build the notebooks:

  • notebook-pr (automatically triggered when a PR is made onto the main repo)

  • publish-book (this needs to be manually triggered by navigating to the github actions tab. See Below) Each workflow involves building python environments and running python scripts on github’s servers, and then automatically creating the appropriate commits and pushes on github. The scripts handle notbeook processing, verification and html generation. After the publish-book workflow is complete, it pushes the generated html files into the gh-pages branch (it creates one if it does not exist). This branch is not to be touched and is automatically taken care of for us. All we need to do is change our pages settings under github settings to use the gh-pages branch instead of main. See Below.

Trigger publish-book:

publish book gif

Point github pages to the gh-pages branch:

github pages gif

Our course content is processed using github’s compute machines. Then we generate the html versions of the course books and host them using github pages. Our scripts push the generated html to the gh-pages branch (do not touch this branch), which is why we need to point the pages setting to use the gh-pages branch.

NOTE: Github supports adding custom urls for you hosted pages course book site. Contact our tech department lead Konstantine Tsafatinos and he will create the appropriate subdomain records for your new course. For example: newcourse.neuromatch.io. Custom domains are delicious, taste some of these:

Enjoy! :)

Summary#

Congrats! You now know:

  1. how and where to create new content

  2. how to follow our naming convention

  3. how to prepare the notebooks for our pipeline

  4. how to create branches and pull requests

  5. the basics of our workflows using github actions

Happy content creating!