• Add default.properties in the library project and add android.library=true in the same
• Run the command in the main Android Project android update project –path . –library ../path-to-the-library/Note: This always has to be relative path.
• Now running ant release should build the library project too.

Note: The android command mentioned in the second step adds an entry in the default.properties file. If you need to override that in different environments, override that in the local.properties. Even there, it has to be relative path, the absolute path will not work.

Generate the build script for test

The suggested practice is to have 2 separate projects for android, one the source and the other for the tests. The following command will generate a build.xml for the test project. Replace the with the path of the source path.

android update test-project -m ../<project-path> -p .

One problem I’ve seen is that, it does not break the build even if there are failures in the test. Issue is reported here:

http://code.google.com/p/android/issues/detail?id=14241

I had to override the run-tests target as mentioned below to fix this issue:

<target name="run-tests" depends="-install-tested-project, install"
description="Runs tests from the package defined in test.package property">
    <echo>Running tests ...</echo>
    <exec executable="${adb}" failonerror="true" outputproperty="tests.output">
        <arg value="shell" />
        <arg value="am" />
        <arg value="instrument" />
        <arg value="-w" />
        <arg value="-e" />
        <arg value="coverage" />
        <arg value="@{emma.enabled}" />
        <arg value="${manifest.package}/${test.runner}" />
    </exec>
    <echo message="${tests.output}"/>
    <fail message="Tests failed!!!">
        <condition>
            <or>
            <contains string="${tests.output}" substring="Error" />
            <contains string="${tests.output}" substring="Fail" />
            </or>
        </condition>
     </fail>
</target>

You can change the ant commands to clean run-tests release in Jenkins to run the tests as part of packaging.

Next I will be writing about how to start emulator from Jenkins while running the tests.

We’ve set up our CI server for building android apps. We use Jenkins as our CI server, but the same steps can be applied to any CI server.

Setup Android Environment on CI server

You need to first install the android SDK and platform tools on the CI server. The steps are well defined here. You can run the command android update sdk --no-ui if the CI server is in an headless environment.

Generate Build script

Using android SDK tool , you can generate build script for the android project which contains the standard steps for building the app such as clean, compile, release, install etc. The following command will generate the build script, replace <appname>, <target> and <project path> accordingly.

android update project -n <appname> -t <target> -p <project directory>

This will create build.xml file under the project directory. You need to create build.properties file with the following contents:

key.store=path-to-keystore
key.alias=[alias]
key.store.password=[pw]
key.alias.password=[pw2]

You can generate the key file using keytoool or you can generate the key file from eclipse. Run the command  ant clean release, which will compile the files, and generate the apk files (it generates signed, unsigned and unaligned files). The signed version can be used for uploading to Android Market or for installing directly on any device. Couple of stuff to be noted here are:

• Ant version has to be 1.8.0 or higher.
• Put the external libraries in the libs directory. Build script automatically picks up the libraries put under libs directory, otherwise the script need to be changed to look at a different classpath.

Checkin the build.xml, build.properties and the key file into the repository.

Setup the CI server

The CI server has to run the ant script for building the app. One more setting what we’ve done in our Jenkins server was to archive the apks as artifacts (available in the post build action). In upcoming posts, I will cover how to do the following:

1. Running android tests
2. Running code/test coverage tools
3. Actual deployment

References: http://skaug.no/ingvald/2011/09/android-app-with-jenkins/

One option that might be required after setting up a build pipeline is the provision to redeploy an earlier version by click of a button. This can become handy when we find a bug in the production and want to see when the bug got introduced. For that we will have to redeploy the earlier version to the staging/UAT environment and test the same.

Jenkins by default does not allow to rerun any of the earlier jobs. But you can achieve this by the Rebuild plugin. It allows you to rebuild any job provided its parameterized build. It also allows you override the parameters. It will show the “Rebuild” button at the left side as shown below when you go to a specific build page:

This is not a replacement for revert build, because it does not take care of reverting DB. But this can be extended to do the same.

Note: It will show the Rebuild button only for those builds which occurred after installing the plugin. I took some time to realize this and it is not very clear in the documentation.

• All the migrations have the down method defined properly. You can check for this by running rake db:migrate:redo
• The schema.rb exists in the repository. This is one of the suggested practices for Rails.

The script is very simple, it extracts the version from the schema.rb and runs the rake db:migrate with the same version. The task gets run automatically along with deploy:rollback. This approach should work for most small and medium complexity Rails apps.

Continued..

• Continuous Delivery – Part 1: Our Jenkins Build Pipeline setup
• Continuous Delivery – Part 2: Code metrics with metrical
• Continuous Delivery – Part 3: Running custom rake tasks during deployment

• Reindex the solr/lucene indexes if any new field has been added to the index
• Some custom rake tasks , say for eg: task to update values in the DB, which you don’t want to add to migrations

We do have Cap tasks for running the required rake tasks, but again it was done manually. So we’ve to remember to run them after deploying to production.  And also there is no way to track what steps were followed for a certain deployment.

I used the build parameter again for fixing the above issue i.e. after accepting the extra tasks as parameter for the build for the first job, they will be passed on to the downstream jobs. In this way the same deployment steps will be automatically followed for rest of the jobs in the pipeline.

The only difference in this case is that – sometimes the parameter can be empty. This check has to be done in the scripts i.e. if the value exists for the parameter, execute the extra tasks along with the normal deployment else just do a normal deployment.

Continued..

• Continuous Delivery – Part 1: Our Jenkins Build Pipeline setup
• Continuous Delivery – Part 2: Code metrics with metrical
• Continuous Delivery – Part 4: Rolling back database migrations with Capistrano rollback

The steps I followed for setting it up in our Jenkins server are:

• Install the gem. I installed it under our ruby 1.9.2 in RVM.
• Create a .metrics file under your projects directory with the settings. This is not mandatory, but I had to use this because some configurations do not seem to be working with ruby 1.9.2. I’ve mentioned those below along with the contents of the .metrics file
• Create a job in Jenkins server, mention the repository details and for build step give the shell command as rvm 1.9.2 -S metrical
• Configure the HTML Publisher plugin for showing the nice Graphs generated by metric_fu as part of the build report. The default report location is tmp/metric_fu/output under the project directory. You can change the same in .metrics file.

As I mentioned above, tools such as flay and flog, which comes as part of metric_fu, have issues working with ruby 1.9.2. And the same case with stats graph and syntax highlighting. The same case with rcov also. So I had to create a .metrics file with the following contents and put in the project dir:

MetricFu::Configuration.run do |config|
        config.code_dirs = ['app', 'lib']
        config.syntax_highlighting = false
        config.metrics  = [:churn,:reek,:roodi,:hotspots,:rails_best_practices]
        config.graphs   = [:reek, :roodi, :rails_best_practices]
        config.rcov[:test_files] = ['spec/**/*_spec.rb']
        config.rcov[:rcov_opts] << "-Ispec" # Needed to find spec_helper
end

For test coverage I’ve used Simplecov which is easy to setup. It will generate the coverage report whenever you run the tests. This also generated html report which can be integrated easily into Jenkins. As mentioned here in the issue list, it does not generate the report when you are running with spork.

Continued..

• Continuous Delivery – Part 1: Our Jenkins Build Pipeline setup
• Continuous Delivery – Part 3: Running custom rake tasks during deployment
• Continuous Delivery – Part 4: Rolling back database migrations with Capistrano rollback

Its a set of python scripts which reverse engineer your server configuration. It stores the configuration locally in its own repository, but you can also:

• Create Puppet/Chef scripts
• Convert into a normal shell script
• Using Blueprint I/O, store the configuration in remote server and pull it when required

It scans all the packages installed on the machine and adds those into the configuration. According to the tutorial, you can do this on a dedicated server or on virtual servers created using VirtualBox etc, or on cloud servers like Amazon EC2, Rackspace etc.

I’ve tried it on a dedicated server (i.e. our CI server) as well as on an EC2 server. I faced a couple of issues, but with their help (see issues 71 and 63), got those fixed.

While bringing up the EC2 instance, you can give the shell script created by Blueprint as user data, provided the size of the script is below 16384 bytes.

Its really an awesome tool. I got the reference to it from Ruby5

• Bring up a small instance
• Install Java
• Convert it into micro instance.

Converting of the instance can be done using API tools, or from the AWS Console. But for me this option was never enabled from the console. I don’t know why. Of course, the API option would still work, I suppose.

But after looking at the bug in detail, I could see a solution which uses the linux nice command, and that did the trick for me. You can see the script as the last comment. The script is as follows:

#!/bin/bash
sudo add-apt-repository "deb http://archive.canonical.com/ natty partner"
sudo apt-get update
#Accept the Java license.
for i in bin jdk jre; do
echo "sun-java6-$i shared/accepted-sun-dlj-v1-1 select true" | sudo debconf-set-selections
done
# convoluted way to install java. this seems to only work some times! Race condition?
# https://forums.aws.amazon.com/message.jspa?messageID=199841#199841
sudo nice --adjustment=19 apt-get install -y sun-java6-jre

The pipeline we have now is very simple. They are:

• Run all the RSpec tests
• Run the Javascript Unit tests
• Deploy to staging
• Deploy to production

The problems we had with our earlier approach, and how the current approach solved those problems:

What we had in the past was, simple scripts with the above items. We had a job for running the tests. If the build was successful, we would run capistrano scripts to deploy to staging and once the customer gave the thumbs up, we’d push it to production.  There are a series of steps that we had to do for each deployment. For eg:  for staging deployment:

• Merge the “staging” branch with master
• Run the cap script
• Tag the build

And almost similar steps needed to be followed for production deployment also. The usual problems that we used to face are:

• There is no way we knew what deployment was done on a certain day for what
• All staging updates would not get pushed to production, so it was a tough task to identify which tag to merge when we were finally ready for a production update

With the current approach, the advantages are:

• Its a one click deployment, so rare chance of making any kind of mistake
• We can clearly see when a feature has been moved to staging, and whether it has been pushed to production or not
• Anyone can do deployment as it is just a click of a button. We may even be able to teach the client how to do this.

One of the major problems I had faced in the earlier “manual” system – was the inability to pass parameters between builds. For example, once the app would be moved to staging, a tag is created and when its ready to be pushed to production, we’d have to merge that particular tag to production and then push it.

To achieve the above I used the “parameterized build” option. In Jenkins, the parameters get converted to Environment Variables and got passed to the downstream job. This is how you can set the parameterized build:

When you fire the build, it prompts for the value of the TAG parameter. Once entered this would be available as an Environment variable for this job and all its downstream jobs.

For eg: in the pipeline, the downstream project is “Stage deploy”, and it will know the value for TAG and uses that value to create a tag in Git after successful deployment.

When it is ready for production deployment it merges with the same tag and deploys to production. In the latest version of the plugin, moving to the next step is not triggered automatically and you can retry the failed ones. The following is a sample pipeline dashboard view:

Note: Capturing the parameter will happen only for the first job. So, if the parameterized job is not the first one in the pipeline, the capture of the parameter will not happen.

So we’ve reached the “one click deployment stage” with this. But we’ve still got the following pending tasks – based on what’s mention in the book  Continuous Delivery:

• Rolling back the bad builds
• Add non-functional tests to the pipeline i.e. performance and security tests
• Feature flags
• Zero downtime deployment
• Canary Releases (Something on the lines of A/B testing)

Continued..

• Continuous Delivery – Part 2: Code metrics with metrical
• Continuous Delivery – Part 3: Running custom rake tasks during deployment
• Continuous Delivery – Part 4: Rolling back database migrations with Capistrano rollback

References:

• http://weblogs.java.net/blog/johnsmart/archive/2011/03/10/build-pipelines-jenkinshudson
• http://continuousdelivery.com/patterns
• http://www.quora.com/How-does-Etsy-manage-development-and-operations