Crazy one line bash script to test web ports

Making a web reminder for this crazy one line bash script checks a web port to see if a server has started up yet. Attempts for ~5 minutes before giving up (Torquebox can take a while to boot).

i=0; while ! (curl --retry 0 --connect-timeout 5 --silent http://localhost:3000 | grep sign_in || false) ; do sleep 5; i=`expr $i + 1`; if [ $i -gt 120 ]; then echo "server failed to respond"; break; else echo "no response. . .yet" done

A valid question would be, Why?. Well, this is useful for the automated build server (Jenkins) that wants to run acceptances tests on a running instance. This means Jenkins needs to wait for the app server to stand up and accept responsibility for its port.

Heck, here is another web reminder for myself, the kill it dead that scans the process list for the pid, but doesnt fail if the process does not exist:

kill -9 `ps aux | grep torquebox | grep java| awk '{print $2}'` || echo "Torquebox was not running"

Fun fact about Rails + Torquebox + Capistrano

If you are getting the error:

/usr/bin/env:
jruby  
No such file or directory

It is due to Torquebox’s JRuby is not in your PATH. You can set this in Capistrano by adding:

set :default_environment, {
    'PATH' => "/path/to/torquebox/jruby/bin:$PATH"
}

RVM vs Torquebox

TIL - When Torquebox cries “no such file to load — torquebox/service_registry” it means it is loading the wrong gem path, which RVM takes control of when it loads. Torquebox wants to run with the jruby runtime it is packaged with, but rvm set the gem paths to its setup.

MRI and JRuby sitting in a tree

If I say KISSing, does that make it a double untundra?

With the power of Bundler’s :platform I have a project with peaceful co-existence between MRI 1.9.3 and JRuby 1.7.3. Everything works in both, devs pick their fancy. The CI runs under JRuby, except we are compiling assets in 1.9.3, and pushes stable builds to a staging server running Torquebox.

gem 'activerecord-jdbcmysql-adapter', "~> 1.2.7", platform: :jruby
gem 'mysql2', "~> 0.3.11", platform: :ruby

group :assets do      
  gem 'therubyrhino', '~> 2.0.2', platform: :jruby
  gem 'therubyracer', '~> 0.11.3', platform: :ruby
end

group :staging, :production do
  gem "torquebox", platform: :jruby 
end

Here is the full Gemfile for the curious.

Easy Obfuscate

It seems APIs are in the air, so far I have rolled out 2 for clients with a third in the works. A recurring issue is what to do with database IDs. There are multiple concerns about exposing DB IDs directly to the API, the big one is inferring the count of a resource. The way around this is to have a random key that is passed around instead of the ID. One way to do this is add an additional column in the database, this always struck me as wasteful. One client it would have involved over 2 million additional fields. It also impacts query perform, as the index is on a varchar instead of a numeric. This is why I started tinkering around with a simple way to obfuscate the IDs instead.

The goal was to create a reasonably sized obfuscated ID so it could be used in an API URL without making it enormous. The obfuscated ID would be converted back into the ID and used to as normal. This is when I stumbled upon Blowfish. Using Blowfish’s block encryption and passing it into a URL encoder will convert an ID up to 99,999,999 into a 12 character string, such as 3NINgAbOhPc=. For brevity, I trim the equals sign that is at the end, so the converted ID is only 11 characters. Here is the Ruby in action:

# Set the salt and Obfuscation mode
Obfuscate.setup do |config|
  config.salt = "A weak salt ..."
  config.mode = :block 
end

obfuscated = Obfuscate.obfuscate( 1 )   # "NuwhZTtHnko"
clarified = Obfuscate.clarify( obfuscated )  # "1"

Now the obfuscated IDs can be exposed by the API instead of the database ID. When an API request passes in an obfuscated ID, it is easily converted to the database ID.

Now all of this wrapped into the obfuscate gem that works with Ruby 1.9 and Rails 3:

https://github.com/mguymon/obfuscate

Chef is painfully awesome

Or how Chef helped me create 100 broken EC2 instances in one day.

The Personal Journey

So, like many, I had the period of private introspect figuring out what infrastructure management tools is for me (more morer morerest). In the end, because Chef hearts Ruby and the Chef ocean is full of Rails fish, I put on my white Toque and dived in.

Hooray it’s Ruby. Oh crap, it’s Ruby

So this ended up being my the biggest problem, with Ruby you can do anything. Which is great, but I want to do a few specific tasks: setup servers for EC2 to run a Rails app. Since Chef is so open ended, there are multiple ways to do the same thing, and most of them are the wrong way. This is particularly hard on beginners, since we won’t identify the bad ideas until they eat our instance first.

Idempotent so easily turns to Idiotpotent

One of the cornerstones for Chef is: Idempotenticy. The management actions (e.g. Recipes packaged in Cookboooks) must be repeatable. A simple example, the first time Nginx recipe runs, it installs Nginx. The second time it runs, if Nginx is installed, nothing happens. The problem is, nothing forces idempotenticy in a recipe, a bad recipe will run fine the first time and then burst into flame the second time. This is less of an issue for mature recipes, but when you get off the beaten trail, Baker beware!

I am my own worse enemy

What really caused me problems was doing something advanced: automatically creating and mounting an EBS share for the Database instances. This caused a bunch of problems, such as:

• Need to format the EBS mount before it can be used
• If EBS share is created before the MySQL recipe, the share permission cannot be set since the MySQL user does not exist yet.
• Ensure the EBS volume is only created once.

I had to make customizations to the Mysql and ElasticSearch recipes to correctly handle using the EBS mount. I am sure a seasoned Chefpert could have created a recipe that used the Mysql and ElasticSearch recipes without customization, but I am happy with the results and it is a solution that we can grow with.

Victory at long last

So in the end, I was able to automate my various servers so that

• Nginx
• RVM
• Elastic Search
• Perconda MySQL
• EBS mounts
• App configuration
• Capistrano deploy

are installed and setup out of the can. It was a bit of trial by fire, I spent a lot time testing various configurations. Once the kinks were finally ironed out and I begin understanding the ins and outs of Chef’s mobius operandi I spent more time creating solutions to my problems than trying to figure out what Chef was doing.

Even though the learning pains of Chef were rough, I cannot imagine going back to the old way of hand built servers. Nothing beats being able to kick an instance out the door with a single command.

Using MRI to precompile Rails’ assets for JRuby

So I do not know how proud I am of this hack, but it does speed up the deploys. Basically, there is a copy the JRuby Gemfile and database.yml as MRI friendly versions. These are used on a MRI setup of Bundler that is used to to run the the rake assets:precompile.

Creating the asset_bundler link to Bundler for Ruby 1.9.3 using RVM:

 rvm wrapper asset_bundler 1.9.3@app bundle

Here is the Capistrano task that handles the precompile on deploy:

namespace :assets do
  task :precompile, :roles => :web, :except => { :no_release => true } do
    run %Q{
      ln -fns #{shared_path}/config/database.mri.yml #{release_path}/config/database.yml &&
      cd #{release_path} && 
      BUNDLE_GEMFILE=AssetGemfile asset_bundle install && 
      echo "Precompiling assets" &&
      BUNDLE_GEMFILE=AssetGemfile asset_bundle exec rake RAILS_ENV=#{rails_env} #{asset_env} assets:precompile
    }
  end
end

The obvious downside is having two Gemfiles, this is mitigated a little by the fact that only the asset scoped gems are being used, so that is the only place that needs to be synced between the two. The asset_bundle install Running the assets:precompile task just needs a valid database.yml as a placeholder, the database is not used.

Things I have learned

Since I get baby pinned in the evening, I have developed quit a few hobbies that only involve one hand. I have gotten pretty good at partial Jazz Hands, Hi-Fives, and Waving. The one handed clapping is still pretty hard.

While I pursue my handy endeavors, I have chew through everything reasonable to watch on NetFlix. This means everything with a combination of Space, Robot, Ninja, and/or Pirate. Now I am left with the horrible burden of turning the TV into something educational. As I start to wade into Documentaries, I must say, there is a woeful lack of Ninja Robot Documentaries.

What I have learned so far:

• There is a Documentary about Fonts, I think it should be filed under Fetishes
• Researchers really like to troll wildlife. No no, it is science, that is why I am teasing him with food.
• Finally, a reasonably priced apt in NYC. The savings really add up when you do not worry about sunlight.

Tales of Upgrades

My attempt to upgrade a Rails 2.3.8 site www.hullbreach.com from 1.9.3 to JRuby 1.6.7. Hopefully this will not turn out to be a horror story. The site is a fairly straight forwards Rails site using twitter-bootstrap: announcements, stories, forums, e.g. nothing crazy going on.

If all goes well, I plan on benching marking the performance of the site for 1.9.3 vs JRuby.

UPDATE - Had to fix a problem with Paperclip’s commandline. The issue is with JRuby not handing Process.spawn correctly.

Finding JRuby friendly Gems

Not everything works with JRuby. Most gems will, any Gem that compiles a C extension will fail right away, but some incompatible gems will not generate an error until runtime. This underscore why it is important to have good test coverage, tests will help suss out misbehaving Gems.

Database

Simply replace the mysql2 gem with activerecord-jdbc-adapter and jdbc-mysql gem in your Gemfile:

gem 'activerecord-jdbc-adapter'
gem 'jdbc-mysql'

Also change the database.yml adapter to mysql (drop the 2):

adapter: mysql

JDBC, you have some explaining to do

When running Webrick, I got this odd error

ActionView::Template::Error (undefined method `explain' for 
#<ActiveRecord::ConnectionAdapters::MysqlAdapter:0x210ab138>):

Apparently, the JDBC for MySQL gem does not support explain that Rails calls in development mode. To get around this, I used an initializer called jdbc_fix.rb with the following:

module ActiveRecord::ConnectionAdapters
  class JdbcAdapter < AbstractAdapter
    def explain(query, *binds)
      ActiveRecord::Base.connection.execute("EXPLAIN #{query}", 'EXPLAIN', binds)
    end
  end
end

This will monkey patch the JdbcAdapter so explain will work with JDBC.

Twitter Bootstrap

Getting twitter-bootstrap-rails to work with JRuby is a pain for several reasons. Right off the bat, it depends on the therubyracers gem. The rubyracers gem in turn depends on the v8 gem, v8 is not JRuby friendly due to the C extensions. The fix for this is using the static branch of gem twitter-bootstrap-rails, which does not use therubyracer and less. Then add therubyrhino to replace therubyracer and ‘less-rails-bootstrap’ for less support. Here is the resulting Gemfile:

 gem 'twitter-bootstrap-rails', 
    :git => "git://github.com/seyhunak/twitter-bootstrap-rails.git", 
    :branch => "static". 
 gem 'less-rails-bootstrap'
 gem 'therubyrhino'

If you get the error:

couldn't find file 'twitter/bootstrap' in application.js

It means you are missing the less-rails-bootrap gem.

Riding the Asset Pipeline, straight to Slow Town

Without the perky v8 gem handling the compilation of assets, the speed of precompiling goes from ~2 minutes to roughly ~5 minutes. Yes, that painful. This is probably the biggest issue for using JRuby with Rails. In development mode and an asset is change, the time to render the page is noticably affected. Needlessly to say, this makes any deployment that involve asset changes a time sink.

Factory Girl loves 1.9

The latest factory_girl only works with 1.9. JRuby by default runs as 1.8. RVM has a couple options to changing the default to 1.9. The other option is to manually set 1.9 via export JRUBY_OPTS=--1.9 or pigging backing on a command line, such as

`JRUBY_OPTS=--1.9 bundle exec`

I suggest using an automated way, it caused me nothing but grief when I was handling it manually.

Paperclip loves 1.9

The latest paperclip requires 1.9. Use the fix for factory_girl. The follow error means that paperclip is failing because the Ruby version is not 1.9

undefined method `set_encoding' for #<Tempfile:0x6c46708a>

If Paperclip spews out the following when saving an attachment

No child processes - No child processes
org/jruby/RubyProcess.java:532:in `waitpid'

You need to fix the command line handler for Paperclip, Cocaine, to avoid the issue of JRuby not handing Process.spawn correctly. To get around this, I used an initializer called cocaine_fix.rb with the following:

Cocaine::CommandLine.runner = Cocaine::CommandLine::BackticksRunner.new

Make sure you use Cocaine 0.3.2, there is an issue with Cocaine 0.4.0 running the identify command.

I never knew how secure the Rails’ console was

Apparently you need jruby-openssl for rails to even start up a console, add the following to your Gemfile:

gem 'jruby-openssl`

Tests

Hooray the tests work! After the gem issues were sorted out, rspec happily ran all the specs, albeit the start up was slow. Setting the Java opts to run in server mode with tierd compilation via export JRUBY_OPTS="--1.9 -J-server -J-XX:+TieredCompilation" seems to make it run faster, but that could just be wishful thinking.

Since the tests are slow to start it, using autotest moves from nice to have to paramount.

Server

I tinkered with a couple servers. The goal was to find a server that could take advantage of the JVM’s multi-threading capabilities.

Good ol’ Webrick

Noticably slower startup when compared with MRI, no suprise there. Once it gets all of its affairs in order, runs snappily. Good for development work, need to find something beefier for production.

Mizuno never made it out of the gate

Starting the Mizuno server with bundle exec mizuno and it spit out this confusing error:

 ClassNotFoundException: Jruby-opensslService

As mentioned eariler, jruby-openssl was installed, and part of the bundle. I pondered and poked this one for a while, even prodded the internet without much luck. I eventually got Mizuno working via

 bundle exec rackup -s mizuno

But I admit I lost faith in mizuno so moved on.

Torquebox

Even with all the trouble I spent getting torquebox to run smoothly, I see so much potential here. Torquebox is a Ruby runtime built on top of JBoss AS. It is a polyglotists dream come true, I want to see worlds collide with Ruby + Clojure + Scala, but that is for another time.

Torquebox is a hefty download at version 2.1.1, with 13 gem dependencies. The torquebox-server gem weighs in at 151M. Woof. Someone needs to be JBoss on a diet. Torquebox was nice enough to work out of the can for my local machine, just have to follow the fairly simple installation instructions.

Getting the needlepoint right

The first thing I did was turn on threading (config.threadsafe!) and torquebox poured cold water on me by locking up. Requests would just hang with no response back. After a bit of fiddling, the issue was class caching. When class caching is enabled, threading works (e.g. product mode), so config.threadsafe! is enabled on production, but disabled for development.

Getting Torquebox out the door

As easy as torquebox was to run locally, it was a pain to setup on my staging server. It took me multiple iterations of deploys and tweaks to my capistrano config to finally get torquebox up and running.

Here is the end result capsitrano settings I used to get torquebox working:

require 'torquebox-capistrano-support'
set :jruby_home,        '<absolute path to jruby>/jruby-1.6.7'
set :torquebox_home,    "<absolute path to gemdir>/torquebox-server-2.1.1-java"
set :jboss_control_style,    :binscripts
set :app_ruby_version,  '1.9' 

I had to deploy once without torquebox-capistrano-support enabled. Then run bundler exec torquebox deploy . to set the rails instance as a torquebox deployment. Finally turn torquebox-capistrano-support back on and deploy.

For the life of me, I could not get torquebox to use Bundler’s deployed gems. Torquebox would start up with an error that it could not successsfully load Bundler.

03:00:36,957 ERROR [org.torquebox.core.runtime] (MSC service thread 1-8) Error 
during evaluation: require %q(bundler/setup): org.jruby.exceptions.RaiseException: 
(GemNotFound) Could not find rake-0.9.2.2 in any of the sources

To get around this, I configured Bundler not use —deploy and used the following override task in the capistrano config:

namespace :bundle do
  task :install do
    run "cd #{release_path} && JRUBY_OPTS='--1.9 -J-XX:+TieredCompilation' 
         bundle install --gemfile #{release_path}/Gemfile --quiet"
  end
end

JRuby was running as 1.8 in Torquebox due to Torquebox’s deployment descriptor were being set to ruby version 1.8. The fix for this is setting the :app_ruby_version in Capistrano.

Once deployed, it took ~30 seconds for torquebox to reload the latest deploy.

Tweaking Torquebox

To get torquebox to run on another port (another Java app already gobbled up 8080) I had to change the socket-binding-name in the torquebox-server-2.1.1-java/jboss/standalone/configuration/standalone.xml:

<socket-binding name='http' port='6060'/>

Starting and Stopping

I used the bin/scripts to start and stop torquebox. I had to manually chmod u+x the torquebox-server-2.1.1-java/jboss/bin/* to make them executable. Unforunately, Torquebox failed to start up with this error:

01:55:02,116 ERROR [org.jboss.as.controller.management-operation] JBAS014612: Operation
("parallel-extension-add") failed - address: ([]): java.lang.RuntimeException: JBAS014670: 
Failed initializing module org.torquebox.core

Luckily I stumbled upon this and set the jruby home in standalone.conf, which allowed Torquebox to start.

APPEND_JAVA_OPTS="-Djruby.home=/path/to/jruby/home"

Finally, victory. Torquebox was beat into submission.

Best Warning Ever:

03:03:01,083 WARN  [org.torquebox.core.as] WARNING: TorqueBox was built and tested with
JRuby 1.6.7.2 and you are running JRuby 1.6.7. **You may experience unexpected results. 
Side effects may include: itching, sleeplessness, and irritability.**

Trinidad

Torquebox soak all my time, Trinidad never got a shot at the lime light.

Final thoughts

I am happy with the end results, I was able to migrate over to JRuby without any loss of functionality. The site is stable without any performance problems. The next step is to run some benchmarks to see how well the site will run under load.