Contributing Guide¶
Attention
A GitHub account is required to begin contributing to AstroConda
Guidelines¶
Attention
The following packaging guidelines are subject to change at any time.
- Please be respectful when commenting on pull-requests or issues.
- If your contribution is not accepted into AstroConda, as a general courtesy, you will be given a clear and concise reason.
- As a contributor you may not claim exclusive rights to a particular recipe.
- You are free to maintain a recipe in AstroConda by issuing regular pull requests.
- Everyone is welcome to improve upon recipes so long as the changes do not introduce packaging conflicts.
- Abandoned recipes may be moved into the
deprecated
directory at any time without warning. (i.e. The package no longer compiles, has been obsoleted, or presents a conflict that cannot be resolved, etc). - Packages derived from
deprecated
recipes will remain available in AstroConda for historical purposes (i.e. to preserve backwards compatibility).
Bugs, questions, and requests¶
Please open a new issue or send us a pull request for bugs, feedback, questions, or enhancements.
- For documentation issues use the astroconda issue tracker
- For recipe issues, use the astroconda-contrib issue tracker
Adding a recipe to astroconda-contrib¶
In this example we will be adding a new recipe to the AstroConda repository for sympy, the symbolic mathematics library.
Navigate to the astroconda-contrib repository on GitHub, login, and create a fork (or click here to have your fork created automatically).
Now that you have a fork of astroconda-contrib, go ahead and clone it to your system:
git clone https://github.com/[Your_Account]/astroconda-contrib
cd astroconda-contrib
To get started adding our recipe, create a new branch and name it sympy-contrib
:
git checkout -t -b sympy-contrib
Git will automatically switch your branch from master
to sympy-contrib
as denoted by the following output:
Branch sympy-contrib set up to track local branch master.
Switched to a new branch 'sympy-contrib'
If you have taken the liberty of looking around the astroconda-contrib directory, you will have noticed a bunch of directories are sitting in there all named by-package. So let’s keep things simple and straight forward. Go ahead and create a directory and name it
sympy
, and proceed inside:
mkdir sympy
cd sympy
Note
This is not an full Conda packaging tutorial. For more information about creating recipes from scratch, please refer to the conda-build documentation.
Hint: Investigate the contents of the recipes in astroconda-contrib. For most cases, copying an existing recipe and changing its values will suffice.
Copy the contents of the astroconda-contrib/template
recipe. Three files bld.bat
, build.sh
, and meta.yaml
will be copied to your working directory:
cp ../template/* .
Go ahead and open meta.yaml
with your favorite plain-text editor:
Caution
It is highly recommended that you enable “tabs to spaces” for your editor. Tab widths are unpredictable and may cause Conda’s YAML parser to fail.
vim meta.yaml
The general structure of the file is as follows:
# These directives are commented here due to Pygments
# failing to parse this section of code.
# ... They are not commented in the real file.
#{% set name = '' %}
#{% set version = '' %}
#{% set number = '0' %}
about:
# Package homepage
home:
# Package license
license:
# A brief description
summary:
package:
# Define these values above
name: {{ name }}
version: {{ version }}
build:
# Define this value above
number: {{ number }}
source:
fn: {{ name }}-{{ version }}.tar.gz
url: http://example.com/example/{{ name }}-{{ version }}.tar.gz
requirements:
build:
# Dependencies required at BUILD-TIME go here
- setuptools
- python x.x
run:
# Dependencies required at RUN-TIME go here
# - ...
#test:
# imports:
# # - (e.g. a_python_module)
#
# commands:
# # - (e.g. program --help)
First, let’s fill in the blanks. Modify the JINJA2 template markers for name
, version
:
{% set name = 'sympy' %}
{% set version = '1.0' %}
Fill in the about
section with relevant information regarding the package:
about:
home: http://sympy.org
license: GPL
summary: Python library for symbolic mathematics
Next, modify the source
section’s url
so that it points to sympy
’s source archive (on the internet):
source:
fn: {{ name }}-{{ version }}.tar.gz
url: https://github.com/{{ name }}/{{ name }}/releases/download/{{ name }}-{{ version }}/{{ name }}-{{ version }}.tar.gz
What’s with the never-ending stream of bracket encapsulated keywords, you ask? Conda uses JINJA2, a basic template system that provides basic string interpolation within recipes. This comes in handy if, let’s say, you decide to build a more recent version of sympy
, you need only modify the first two variable definitions in this file (assuming the URL structure has not changed).
The requirements
section may be confusing to some, so let’s clarify the distinction between build
and run
before diving in. The build
section defines Conda packages required at compile-time (i.e. python setup.py install
), whereas the run
section lists Conda packages required at install-time (i.e. conda install [package]
).
As recipe maintainer the method used to dependency discover is that of trial and error. For many Python packages obtained via PyPi, it is easy enough to visually examine setup.py
or requirements.txt
to get a good idea of the recipes you need to depend on. Some package may require several iterations of executing conda build
and returning to your recipe in the editor to append more packages.
As we can see below, sympy
requires mpmath
, setuptools
and python
to build the recipe, but only mpmath
and python
to run it successfully after installation:
requirements:
build:
- mpmath
- setuptools
- python x.x
run:
- mpmath
- python x.x
What does the x.x
imply exactly? This instructs conda build
not to proceed unless python=[version]
has been issued as an argument on the command-line. If x.x
is omitted here, the recipe will choose the version of Python currently active in your environment. In most cases it is best to be explicit rather than implicit when it comes to defining version requirements in Conda.
The test
section defines few useful lists, imports
, commands
, and requires
. While these are not required to be used in any given recipe, we do use them in AstroConda. The imports
section is a list of Python module imports, the commands
are executed in a basic shell environment, and finally requires
defines any extraneous packages to be installed into the environment before running the tests.
test:
imports:
- sympy
#commands:
# - no shell commands to execute
#requires:
# - does not require any extra testing-related packages
If sympy
provided a command-line utility named sympy-show
, you would use the commands
section to verify the utility’s functionality. A simple example of this would be to output a usage statement.
test:
# ...
commands:
- sympy-show --help
If a program returns greater than zero conda build
will fail as if it observed an error. Not all programs return zero after issuing --help
(or an equivalent argument). Due to this, you may need to omit this style of test.
It will not hurt to keep the commands
section populated but disabled with a note describing why it doesn’t work. Others will find this information useful. Given this scenario, the optimal approach would be to contact the developers and plead with them to
normalize the exit value.
Below is our sympy
final recipe. Despite the overwhelming use of JINGA2 in our example, things are looking pretty streamlined.
{% set name = 'sympy' %}
{% set version = '1.0' %}
{% set number = '0' %}
about:
home: http://sympy.org
license: GPL
summary: Python library for symbolic mathematics
source:
fn: {{ name }}-{{ version }}.tar.gz
url: https://github.com/{{ name }}/{{ name }}/releases/download/{{ name }}-{{ version }}/{{ name }}-{{ version }}.tar.gz
requirements:
build:
- mpmath
- setuptools
- python x.x
run:
- mpmath
- python x.x
test:
imports:
- sympy
The template
directory copied earlier in this tutorial contains two basic python build scripts for both *Nix (build.sh
) and Windows (bld.bat
), and is coincidentally compatible with the example we’re using here. Not all Python packages (especially Makefile-based packages) will compile successfully using this “one-liner” template. Always refer to the INSTALL
file or equivalent documentation for the program you are attempting to compile to learn more about what the package expects from the end-user at compile-time.
Before we can issue a pull request on GitHub, we first ensure it builds, tests, and installs properly on our local system. To do this we need to change our directory to one level above the recipe.
cd ..
# i.e. /path/to/astroconda-contrib
Now run conda build
to compile our sympy
recipe into a Conda package. In the example below we are building against
Python 3.5:
conda build -c http://ssb.stsci.edu/astroconda --skip-existing --python=3.5 sympy
That’s probably a bit more involved than you thought. Let’s break it down. We issue -c [URL]
which instructs the build to utilize the AstroConda channel while checking for package dependencies (i.e. the recipe’s requirements
section). Secondly, we issue --skip-existing
to prevent conda build
from rebuilding dependencies discovered in the local astroconda-contrib directory. That is to say, if a package defined as a requirement exists remotely, it will then download and install it, rather than rebuild it from scratch. --python=
is self-explanatory, and the final argument is the name of the recipe(s) we intend to build.
At this point, if the build was successful, our Conda package (a bzipped tarball) called sympy-1.0-py35_0.tar.bz2
is emitted to /path/to/miniconda3/conda-bld/[os-arch]/
. This directory is a local Conda package repository.
To install this new sympy
package and interact with it ourselves you could run the following:
conda create -n sympy_test --use-local sympy
source activate sympy_test
Then manually verify the package is working:
python
And checking it out for yourself:
>>> import sympy
>>> sympy.__file__
'/path/to/miniconda3/envs/sympy_test/lib/python3.5/site-packages/sympy/__init__.py'
Now that you have verified the recipe is fully functional and are happy with the outcome, it’s time to create a pull request against astroconda-contrib main repository.
Push your sympy-contrib
branch up to your fork on GitHub:
git push origin sympy-contrib
It is recommended that you familiarize yourself with GitHub pull requests before proceeding (see: tutorial).
Using GitHub, you will want to browse to your astroconda-contrib
fork, select the sympy-contrib
branch from
the drop-down menu (the default will read: “Branch: master”, next to a black downward-pointing caret). Once selected, click the large green button labeled: “New pull request”.
From here, you may wish to edit the title of your pull request and add initial comments or notes regarding what you have done, or list any work that may still need to be done. After submitting your pull request, a member of the Science Software Branch at STScI, or fellow contributors will review the requested changes, ask questions, offer feedback or advice.
At this point if everything appears to be in order your recipe will likely be merged, built, and incorporated into AstroConda!
Updating a recipe in astroconda-contrib¶
Let’s assume time has passed and our sympy
package from the previous example is no longer up to date with what’s generally available on GitHub. Updating recipes is a fairly straight forward process.
At the top of the file you will remember we have a few variables defined encapsulated by {% %}
. These jinja2
variables control the name, version, and build revision of the recipe. Using variable interpolation saves time, because it’s much easier to edit a single variable that effects an entire recipe, than it is to search/replace specific fields. Typos are also much easier to spot.
{{ name }}
, {{ version }}
and {{ number }}
expand to sympy
, 1.0
and 0
respectively:
{% set name = 'sympy' %}
{% set version = '1.0' %}
{% set number = '0' %}
[...]
build:
number: {{ number }}
[...]
source:
fn: {{ name }}-{{ version }}.tar.gz
url: https://github.com/{{ name }}/{{ name }}/releases/download/{{ name }}-{{ version }}/{{ name }}-{{ version }}.tar.gz
So to update sympy
to version 1.2
, for example, you would perform the following steps in your forked astroconda-contrib
repository.
Checkout a new branch¶
git checkout -tb update-sympy master
Doing this ensures your new branch is based on master
rather than your current branch, if any. It also keeps your master
branch pristine, avoiding merge conflicts in the future.
Make your modifications¶
$EDITOR sympy/meta.yaml
[...]
{% set version = '1.2' %}
# ^ Was '1.0', but not anymore.
Now save and exit your editor.
Review your modifications¶
As stated earlier, the fastest way to find out whether your recipe works correctly is to try building it for yourself.
conda build -c http://ssb.stsci.edu/astroconda --skip-existing --python=2.7 sympy
conda build -c http://ssb.stsci.edu/astroconda --skip-existing --python=3.5 sympy
Did it work? If not, review the error message and make changes accordingly.
Commit your modifications¶
Assuming you are able to build the package locally, then you’re ready to push your changes up to your fork.
git add sympy/meta.yaml
git commit -m 'Update sympy 1.0 -> 1.2'
git push origin update-sympy
Open a pull request¶
See: Using Pull Requests
- Using your browser, visit the
update-sympy
branch of your fork: https://github.com/YOUR_ACCOUNT/astroconda-contrib/tree/update-sympy - Click the gray “New pull request” button
- Fill out the pull request form
- Click the green “Create pull request” button
That’s all there is to it. One of our maintainers will review the pull request and get back to you.