Sketchpad - Dan Sketcher's personal blog

We use Capistrano exclusively for deployment to live and pre-live environments, so imagine my surprise when, copying a configuration into files on a new app, that I started seeing this:

Flexo:app dansketcher$ cap production misc:show_path
    triggering load callbacks
  * 2013-01-31 13:17:53 executing `production'
    triggering start callbacks for `misc:show_path'
  * 2013-01-31 13:17:53 executing `multistage:ensure'
  * 2013-01-31 13:17:53 executing `misc:show_path'
  * executing "echo $PATH"
`misc:show_path' is only run for servers matching {}, but no servers matched

The ‘production’ stage was configured, but even adding things like raise '' in the config/deploy/production.rb file failed to have an effect. It was like the file wasn’t even being loaded, except that trying to execute against a non-existent stage (‘work_damn_you’ stage, anyone?) did raise an error.

In the end, out of desperation, I ran dos2unix on the files, which weirdly complained about a binary file. Hrm. I re-saved the files as UTF-8 and all started working.

So be warned.

Having provisioned a new server, I was updating my Capistrano deploy.rb to handle the new box, and on testing a new deploy, I was getting this:

** [server :: out] == Unfgit: =============================================================
** [server :: out] Permission denied public key error. account: xxxxx; repository: xxxxx; person: none; message: no person found on this account using this public key
** [server :: out] ========================================================================

Annoying, seeing as it worked just find on the old box.

I went onto the machine and confirmed using the ssh key there I could check out the repo properly. Unfuddle support tried to help but could not. I fortunately came across this github resource which explains some of the settings. I think on the third or so re-reading I noticed this real impact of this

Agent Forwarding

If you’re using your own private keys for git you might want to tell Capistrano to use agent forwarding with this command. Agent forwarding can make key management much simpler as it uses your local keys instead of keys installed on the server.

ssh_options[:forward_agent] = true

So, I tried removing that line from my cap deploy.rb, and the checkout worked fine.

Hope this helps someone else!

Success! I have managed to get Splunk to group Rails logs into logical groups.

First, I changed my Rails web server to be a light forwarder, as I wanted the indexer to do the grouping (as per the behaviour breakdown from the previous part of this series). Then:

$SPLUNK_HOME/etc/system/local/inputs.conf

[default]
host = web.server.example.com

[monitor:///var/www/sites/web.application.example.com/shared/log/production.log*]
disabled = false
followTail = 0
sourcetype = rails_log

[monitor:///var/log/httpd]
disabled = false
followTail = 0
sourcetype = access_combined

I’m not sure if I need to have the [rails_log] declaration on the LightForwarder’s etc/system/local/props.conf file, but it’s there as an empty declaration just in case.

Before we go further, note that my Rails logging uses log4r and a custom formatter, so my logs look like this:

[ApplicationName][INFO] 100819 06:37:07 :: 

Processing SessionsController#new (for 172.0.0.1 at 2010-08-19 06:37:07) [GET]
[ApplicationName][INFO] 100819 06:37:07 ::   Parameters: {"unauthenticated"=>"true", "action"=>"new", "controller"=>"sessions"}
[ApplicationName][INFO] 100819 06:37:07 :: Rendering template within layouts/agent
[ApplicationName][INFO] 100819 06:37:07 :: Rendering sessions/new
[ApplicationName][INFO] 100819 06:37:07 :: Completed in 20ms (View: 12, DB: 5) | 200 OK [https://web.application.example.com/user/sign_in?unauthenticated=true]
[ApplicationName][INFO] 100820 01:01:25 :: 

Processing Index#index (for 172.0.0.1 at 2010-08-20 01:01:25) [GET]
[ApplicationName][INFO] 100820 01:01:25 ::   Parameters: {"action"=>"index", "controller"=>"indes"}
[ApplicationName][INFO] 100820 01:01:25 :: Filter chain halted as [:authenticate] rendered_or_redirected.
[ApplicationName][INFO] 100820 01:01:25 :: Completed in 3ms (View: 1, DB: 4) | 401 Unauthorized [https://web.application.example.com/]

The important thing to note here is that Rails inserts 2 linebreaks before putting “Processing blah blah” so we’ll use that to split. In the props.conf documentation, it recommends using the LINE_BREAKER item to handle multiline breaks, and it seemed easier to me as it uses just one regexp to break the input stream. The important part for this kind of splitting is to know:

* Wherever the regex matches, the start of the first matching group is considered the end of the 
  previous event, and the end of the first matching group is considered the start of the next event.
* The contents of the first matching group is ignored as event text.

What this means is that the LINE_BREAKER regexp (using PCRE) must have a match group that can be discarded for matching. If you grep over the default provided files (`grep -Rn “LINE_BREAKER” /opt/splunk/`), you’ll see that most of the options use a linebreak or multiples thereof as the first match group. We’ll do the same. Other than that, we just want to match on the “Processing” text as the start delimiter… which comes out to the below…

On the indexer’s $SPLUNK_HOME/etc/system/local/props.conf

[rails_log]
LINE_BREAKER = ([\r\n]).* \:\: [\r\n]+Processing

Now, on the index, the rails_log items are multiline events!

Edit 2010-03-17: Harder than it first seemed! That permission only allows Apache to load Passenger. See here for the full requirements

I’ve just started playing with SELinux on Centos, and while the idea is great, it’s not exactly what I’m used to. Take for example adding Phusion Passenger to Apache. When I first restarted, I got this

[root@localhost modules]# /etc/init.d/httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd: httpd: Syntax error on line 210 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.d/passenger.conf: Cannot load /opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/passenger-2.2.9/ext/apache2/mod_passenger.so into server: /opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/passenger-2.2.9/ext/apache2/mod_passenger.so: failed to map segment from shared object: Permission denied
                                                           [FAILED]

Huh?

So it turns out after reading through some google results and then subsequently man httpd_selinux that the following will fix it:

chcon -R -h -t httpd_sys_script_exec_t /opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/passenger-2.2.9/ext/apache2/mod_passenger.so

And so it does. “httpd_sys_script_exec_t” allows Apache to execute the SO, and we’re all good to restart.