<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[Wallaby updates]]></title>
  <link href="http://getwallaby.com/atom.xml" rel="self"/>
  <link href="http://getwallaby.com/"/>
  <updated>2013-01-24T10:25:02-06:00</updated>
  <id>http://getwallaby.com/</id>
  <author>
    <name><![CDATA[William Benton]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[Authorization for Wallaby clients]]></title>
    <link href="http://getwallaby.com/2012/09/authorization-for-wallaby-clients/"/>
    <updated>2012-09-12T17:30:00-05:00</updated>
    <id>http://getwallaby.com/2012/09/authorization-for-wallaby-clients</id>
    <content type="html"><![CDATA[<p>Wallaby 0.16.0, which updates the Wallaby API version to 20101031.6, includes support for authorizing broker users with various roles that can interact with Wallaby in different ways.  This post will explain how the authorization support works and show how to get started using it.  If you just want to get started using Wallaby with authorization support as quickly as possible, skip ahead to the section titled &#8220;Getting Started&#8221; below.  Detailed information about which role is required for each Wallaby API method is <a href="http://getwallaby.com/api-roles/">available here</a>.</p>

<h2>Overview</h2>

<p>Users must authenticate to the AMQP broker before using Wallaby (although some installations may allow users to authenticate as &#8220;anonymous&#8221;), but previous versions of Wallaby implicitly authorized any user who had authenticated to the broker to perform any action.  Wallaby now includes a database mapping from user names to <em>roles</em>, which allows installations to define how each broker user can interact with Wallaby.  Each method is annotated with the role required to invoke it, and each method invocation is checked to ensure that the currently-authenticated user is authorized to assume the role required by the method.  The roles Wallaby recognizes are <code>NONE</code>, <code>READ</code>, <code>WRITE</code>, or <code>ADMIN</code>, where each role includes all of the capabilities of the role that preceded it.</p>

<p>If <code>WALLABY_USERDB_NAME</code> is set in the Wallaby agent&#8217;s environment upon startup and represents a valid pathname, Wallaby will use that as the location of the user-role database.  If this variable is set to a valid pathname but no file exists at that pathname, the Wallaby user-role database will be created upon agent startup.  If <code>WALLABY_USERDB_NAME</code> is not set, the user-role database will be initialized in memory only and thus will not persist across agent restarts.</p>

<h3>Standard authorization</h3>

<p>When Wallaby is about to service an API request, it:</p>

<ol>
<li><p>checks the role required to invoke the method.</p></li>
<li><p>checks the authorization level specified for the user.  There are several possibilities under which a user could be authorized to invoke a method:</p>

<ul>
<li>the user is explicitly authorized for a role that includes the required role (e.g. the user has an <code>ADMIN</code> role but the method only requires <code>READ</code>);</li>
<li>the user is implicitly authorized for a role that includes the required role (e.g. there is an entry for the wildcard user <code>*</code> giving it <code>READ</code> access and the method requires <code>READ</code> access)</li>
<li>the role database is empty, in which case all authenticated users are implicitly authorized for all actions (this is the same behavior as in older versions of Wallaby)</li>
<li>the invocation is of a user-role database maintenance method and the client is authorized via shared secret (see below)</li>
</ul>
</li>
<li><p>if none of the conditions of the above step hold, the method invocation is unauthorized and fails with an API-level error.  If the API method is invoked over the Ruby client library, it will raise an exception.  If it is invoked via a <code>wallaby</code> shell command-line tool, it will print a human-readable error message and exit with a nonzero exit status.</p></li>
<li><p>if the user is authorized to invoke the method, invocation proceeds normally.</p></li>
</ol>


<h3>Authorization with secret-based authentication</h3>

<p>This version of the Wallaby API introduces three new methods:  <code>Store#set_user_privs</code>, <code>Store#del_user</code>, and <code>Store#users</code>.  These enable updating and reading the user-role database; the first two require <code>ADMIN</code> access, while the last requires <code>READ</code> access.  Because changes in the user-role database may result in an administrator inadvertently removing administrator rights from his or her broker user, Wallaby provides another mechanism to authorize access to these methods.  Each of these three methods supports a special <code>secret</code> option in its <code>options</code> argument.  When the Wallaby service starts up, it loads a secret string from a file.  Clients that supply the correct secret as an option to one of these calls will be authorized to invoke these calls, even if the broker user making the invocation is not authorized by the user-role database.</p>

<p>The pathname to the secret file is given by the environment variable <code>WALLABY_SECRET_FILE</code>.  If this variable is unset upon agent startup, Wallaby will not use a shared secret (and secret-based authorization will not be available to API clients).  It this variable is set and names an existing file that the Wallaby agent user can read, the Wallaby shared secret will be set to the entire contents of this file.  If this variable is set and names a nonexistent file in a path that does exist, Wallaby will create a file at this path upon startup with a randomly-generated secret (consisting of a digest hash of some data read from <code>/dev/urandom</code>).  If this variable is set to a pathname that includes nonexistent directory components, the Wallaby agent will raise an error.  If you create your own secret file, ensure that it is only readable by the UNIX user that the Wallaby agent runs as (typically <code>wallaby</code>).</p>

<h3>Caveats</h3>

<p>The Wallaby agent&#8217;s authorization support is designed to prevent broker users from altering Condor pool configurations in excess of their authority.  It is not intended to keep all configuration data strictly confidential.  (This is not as bad as it might sound, since Wallaby-generated configurations are available for inspection by Condor users.)  Furthermore, due to technical limitations, it is not possible to protect object property accesses over the API with the same authorization support that we use for API method invocations.  Therefore, if concealing configuration data from some subset of users is important for your installation, you should prevent these users from authenticating to the broker that the Wallaby agent runs on.</p>

<h2>Getting started</h2>

<p>Here is a quick overview of how to get started with auth-enabled Wallaby:</p>

<ol>
<li>Stop your running Wallaby and restart your broker before starting the new Wallaby (this is necessary to pick up the new API methods).  Set <code>WALLABY_USERDB_NAME</code> in your environment to a path where you can store the user-role database.  Install and start your new Wallaby.</li>
<li>If you&#8217;re using the RPM package, it will create a &#8220;secret file&#8221; for you in <code>/var/lib/wallaby/secret</code>.  If not, you will need to set <code>WALLABY_SECRET_FILE</code> in the environment to specify a location for this secret file and then restart Wallaby.  The Wallaby secret is a special token that can be passed to certain API methods (specifically, those related to user database management) in order to authorize users who aren&#8217;t authorized in the user database.</li>
<li>Try using some of the new shell commands:  <code>wallaby set-user-role</code>, <code>wallaby list-users</code>, and <code>wallaby delete-user</code>.</li>
<li>Make sure that you have a secret in your secret file.  Make a note of it.  Try setting the role for your current broker user to <code>READ</code> or <code>NONE</code> (e.g. &#8220;wallaby set-user-role anonymous NONE&#8221;) and then see what happens when you try and run some other Wallaby shell commands.  You can recover from this by passing the Wallaby secret to &#8220;wallaby set-user-role&#8221;; see its online help for details.</li>
</ol>


<p>The default user database is empty, which will result in the same behavior as in older versions of Wallaby (viz., all actions are available to all broker users), but only until a user role is added, at which point all actions must be explicitly or implicitly authorized.</p>

<p><em>This article is cross-posted from <a href="http://chapeau.freevariable.com/2012/09/authorization-for-wallaby-clients.html">Chapeau</a></em></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Highly-available configuration data with Wallaby]]></title>
    <link href="http://getwallaby.com/2012/08/live-backup/"/>
    <updated>2012-08-29T09:40:00-05:00</updated>
    <id>http://getwallaby.com/2012/08/live-backup</id>
    <content type="html"><![CDATA[<p>Many Condor users are interested in <em>high-availability</em> (HA) services:  they don&#8217;t want their compute resources to become unavailable due to the failure of a single machine that is running an important Condor daemon.  (See <a href="http://research.cs.wisc.edu/condor/CondorWeek2012/presentations/rati-benton-condor-ha.pdf">this talk</a> that Rob Rati and I gave at Condor Week this year for a couple of solutions to HA with the Condor <code>schedd</code>.)  So it&#8217;s only natural that Condor users who are interested in configuring their pools with <a href="http://getwallaby.com">Wallaby</a> might wonder how Wallaby responds in the face of failure.</p>

<p>Some of the technologies that the current version of Wallaby is built upon do not lend themselves to traditional active-active high-availability solutions.  However, the good news is that due to Wallaby&#8217;s architecture, almost all running Condor nodes <em>will not be affected</em> by a failure of the Wallaby service or the machine it is running on.  Nodes that have already checked in with Wallaby will have their latest activated configurations as of the last checking.  The only limitations in the event of service failure are:</p>

<ol>
<li>new nodes will not be able to check in and get the default configuration;</li>
<li>it will not be possible to access older activated configurations; and</li>
<li>it will not be possible to alter, activate, or deploy the configuration.</li>
</ol>


<p>These limitations, of course, disappear when the service is restored.  For most users, losing the ability to update or deploy configurations due to a service failure — but not losing their deployed configurations or otherwise affecting normal pool operation — represents an acceptable risk.  Some installations, especially those who aggressively exploit Wallaby&#8217;s scripting interface or versioning capability, may want a more robust solution:  these users might want to be able to access older versions of their activated configurations even if Wallaby is down, or they might want a mechanism to speed recovery by starting a replica of their service on another machine.  In the remainder of this post, we&#8217;ll discuss some approaches to provide more access to Wallaby data when Wallaby is down.</p>

<h3>Accessing older versioned configurations</h3>

<p>If you need to access historical versioned configurations, the easiest way to do it is to set up a <code>cron</code> job on the machine running Wallaby that periodically runs <code>wallaby vc-export</code> on your snapshot database and outputs versioned configurations to a shared filesystem.  <code>wallaby vc-export</code>, which is documented <a href="http://getwallaby.com/2011/11/exporting-versioned-configurations/">in this post</a>, exports all historical snapshots to plain-text files, so you can access the configuration for <code>foo.example.com</code> at version 1234 in a file called something like <code>1234/nodes/foo.example.com</code>.  This <code>cron</code> job needs to be able to access the filesystem path of the Wallaby snapshot database; furthermore, to run it efficiently, you&#8217;ll probably want to limit the number of snapshots it processes each time; see <code>wallaby vc-export</code>&#8217;s online help for more details.</p>

<h3>Exporting Wallaby state to a file</h3>

<p>If you&#8217;re just interested in the state of the Wallaby service (including possibly unactivated changes), you can periodically run <code>wallaby dump</code> over the network.  This will produce a YAML file consisting of the serialized state of the Wallaby; you can later load this file by using the <code>wallaby load</code> command, possibly against another Wallaby agent.</p>

<h3>Backing up the raw database files</h3>

<p>The easiest way to pick up and recover from a Wallaby node failure is to start a new Wallaby service with the same databases as the failed node.  In turn, the easiest way to do this is by periodically copying these database files from their locations on the Wallaby node (typically in <code>/var/lib/wallaby</code>) to some location on a shared filesystem.  The following Ruby script<noscript>, which you&#8217;ll need to view this post in a browser and not a feed reader to see,</noscript> will safely copy the SQLite files that Wallaby uses even while Wallaby is running:</p>

<figure class='code'><figcaption><span>backup-db.rb </span><a href='https://gist.github.com/3518892'>script for live backup of Wallaby databases</a></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
</pre></td><td class='code'><pre><code class='ruby'><span class='line'><span class="c1">#!/usr/lib/env ruby</span>
</span><span class='line'>
</span><span class='line'><span class="c1"># Acquires a shared lock on a SQLite database file and copies it to a backup</span>
</span><span class='line'><span class="c1"># usage:  backup-db.rb DBFILE.db BACKUPFILE.db</span>
</span><span class='line'><span class="c1"># author:  William Benton (willb@redhat.com)</span>
</span><span class='line'><span class="c1"># Public domain.</span>
</span><span class='line'>
</span><span class='line'><span class="nb">require</span> <span class="s1">&#39;sqlite3&#39;</span>
</span><span class='line'><span class="nb">require</span> <span class="s1">&#39;fileutils&#39;</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">backup_db</span><span class="p">(</span><span class="n">db_file</span><span class="p">,</span> <span class="n">backup_file</span><span class="p">)</span>
</span><span class='line'>  <span class="k">begin</span>
</span><span class='line'>    <span class="n">db</span> <span class="o">=</span> <span class="no">SQLite3</span><span class="o">::</span><span class="no">Database</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">db_file</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">db</span><span class="o">.</span><span class="n">transaction</span><span class="p">(</span><span class="ss">:immediate</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">ignored</span><span class="o">|</span>
</span><span class='line'>      <span class="c1"># starting a transaction with &quot;:immediate&quot; means we get a shared lock</span>
</span><span class='line'>      <span class="c1"># and thus any db writes (unlikely!) complete before we copy the file</span>
</span><span class='line'>      <span class="no">FileUtils</span><span class="o">.</span><span class="n">cp</span><span class="p">(</span><span class="n">db_file</span><span class="p">,</span> <span class="n">backup_file</span><span class="p">,</span> <span class="ss">:preserve</span><span class="o">=&gt;</span><span class="kp">true</span><span class="p">)</span>
</span><span class='line'>    <span class="k">end</span>
</span><span class='line'>  <span class="k">ensure</span>
</span><span class='line'>    <span class="n">db</span><span class="o">.</span><span class="n">close</span>
</span><span class='line'>  <span class="k">end</span>
</span><span class='line'><span class="k">end</span>
</span><span class='line'>
</span><span class='line'><span class="n">backup_db</span><span class="p">(</span><span class="no">ARGV</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span><span class="p">,</span> <span class="no">ARGV</span><span class="o">[</span><span class="mi">1</span><span class="o">]</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>(cross-posted from <a href="http://chapeau.freevariable.com/2012/08/highly-available-configuration-data-with-wallaby.html">Chapeau</a>)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Using the skeleton group]]></title>
    <link href="http://getwallaby.com/2012/06/using-the-skeleton-group/"/>
    <updated>2012-06-15T12:46:00-05:00</updated>
    <id>http://getwallaby.com/2012/06/using-the-skeleton-group</id>
    <content type="html"><![CDATA[<p>In Wallaby, Condor nodes are configured by applying <em>features</em> and <em>parameter</em> settings to <em>groups</em>.  In order for the group abstraction to be fully general, <a href="http://getwallaby.com/2011/05/using-wallaby-groups-to-implement-node-tagging/">Wallaby provides two kinds of <em>special groups</em></a>:  the <em>default group</em>, which contains every node (but which is the lowest-priority membership for each node), and a set of <em>identity groups</em>, each of which only contains a single node (and which is always its highest-priority membership, so that special settings applied to a node&#8217;s identity group always take precedence over settings from that node&#8217;s other memberships).</p>

<p>The default group provides a convenient mechanism to apply some configuration to every node in a pool, whether or not it has been explicitly configured; this enables us to provide some sensible basic configuration for nodes that are added to the pool opportunistically or from the cloud.  However, for some applications, using the default group may be inflexible because of Wallaby&#8217;s additive, compositional configuration model.  For example, what if you want to enable some functionality by default but disable it on some subset of nodes?  Or what if you want to use &#8220;defaults&#8221; to override some settings made in explicit groups?</p>

<p>The <em>skeleton group</em> provides a solution to these problems.  (The &#8220;skeleton group&#8221; is so named by analogy with the <a href="http://www.linuxhowtos.org/Tips%20and%20Tricks/using_skel.htm"><code>/etc/skel</code></a> directory on Linux systems, which provides a template for the home directories of newly-created users.)  Like the default group, every new node is added to the skeleton group when it is created and receives the last-activated configuration for the skeleton group. Unlike the default group, individual nodes&#8217; skeleton group memberships can be at different priorities; furthermore, nodes can be removed from the skeleton group all together.  In this post, we&#8217;ll see how to set it up and use it.</p>

<ol>
<li>Ensure that you&#8217;re running Wallaby 0.15.0 or later.  (Some older versions also include experimental support for the skeleton group, but Wallaby 0.15.0 eliminates many rough edges and is the recommended minimum version for using the skeleton group.)</li>
<li>Change the environment that your <code>wallaby-agent</code> runs in so that <code>WALLABY_ENABLE_SKELETON_GROUP</code> is set to <code>true</code>.  The easiest way to do this is to modify the <code>/etc/sysconfig/wallaby-agent</code> file (on RHEL 5 and 6) or the <code>/etc/sysconfig/wallaby-agent-env</code> file (on Fedora).</li>
<li>Restart Wallaby by running <code>service wallaby restart</code> (on RHEL 5 and 6) or <code>systemctl  restart wallaby.service</code> (on Fedora).</li>
<li>Create a new node, either explicitly (e.g. <code>wallaby add-node foo.local.</code>) or by having a new node check in; note that its memberships include <code>+++SKEL</code>.  This is the skeleton group.  (Preexisting nodes won&#8217;t be in the skeleton group by default, but it&#8217;s straightforward to add them all via the Wallaby shell.)</li>
</ol>


<p>Now that you have it set up, here are some things to try doing with it:</p>

<ol>
<li>Put some of your default-group configuration in the skeleton group.  Note that the skeleton group will be validated in isolation just as the default group is, but the validation procedure for the skeleton group assumes that the default group configuration has also been applied already &#8212; so your skeleton group configuration will need to be valid when applied atop the default group configuration.  (This is so that newly-created nodes will have a valid configuration.)</li>
<li>Put configuration that you want to override on some nodes in the skeleton group.  Then remove these nodes from the skeleton group.</li>
<li>Experiment with configurations that depend on the priority of the skeleton group; move it around in nodes&#8217; membership lists.</li>
<li><a href="http://getwallaby.com/2010/10/extending-the-wallaby-shell/">Write a Wallaby shell command</a> to copy the configuration from the skeleton group over to a node&#8217;s identity group and then remove that node from the skeleton group (in order to allow further customization).</li>
</ol>


<p>Let us know how you wind up using this functionality!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Troubleshooting Condor with Wallaby]]></title>
    <link href="http://getwallaby.com/2012/06/troubleshooting/"/>
    <updated>2012-06-01T12:27:00-05:00</updated>
    <id>http://getwallaby.com/2012/06/troubleshooting</id>
    <content type="html"><![CDATA[<p>Often, if you&#8217;re trying to reproduce a problem someone else is having with Condor, you&#8217;ll need their configuration.  Likewise, if you&#8217;re trying to help someone reproduce a problem you&#8217;re having, you&#8217;ll want to send along your configuration to aid them in replicating your setup.  For installations that use legacy flat-file configurations (optionally with a local configuration directory), this can be a pain, since you&#8217;ll need to copy several files from site to site (ensuring that you&#8217;ve included all the files necessary to replicate your configuration, perhaps across multiple machines on the site experiencing the problem).</p>

<p>If everyone involved uses <a href="http://getwallaby.com/">Wallaby</a> for configuration management, things can be a lot simpler:  the site experiencing the problem can use <code>wallaby dump</code> to save the state of their configuration for an entire pool to a flat file, which the troubleshooting site can then inspect or restore with <code>wallaby load</code>.  If the problem appears in some configuration snapshots but not in others, the reporting site can use <a href="http://getwallaby.com/2011/11/exporting-versioned-configurations/"><code>wallaby vc-export</code></a> to generate a directory of <em>all</em> of their configurations over time, so that the troubleshooting site can attempt to pinpoint the differences between what worked and what didn&#8217;t.</p>

<p>(Thanks to <a href="http://spinningmatt.wordpress.com">Matt</a> for pointing out the value of versioned semantic configuration management in reproducing problems!)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Now powered by OctoPress]]></title>
    <link href="http://getwallaby.com/2012/03/getwallaby-dot-com-is-now-powered-by-octopress/"/>
    <updated>2012-03-16T12:27:00-05:00</updated>
    <id>http://getwallaby.com/2012/03/getwallaby-dot-com-is-now-powered-by-octopress</id>
    <content type="html"><![CDATA[<p>I noticed yesterday that this site had recently fallen victim to yet another WordPress security vulnerability; please accept my apologies if you&#8217;ve seen any unexpected behavior here.  In other news, I&#8217;m pleased to announce that getwallaby.com is now powered by <a href="http://octopress.org">OctoPress</a>.  Please <a href="https://github.com/willb/getwallaby.com/issues">let me know</a> if you find any quirks with the new site.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Exporting versioned configurations]]></title>
    <link href="http://getwallaby.com/2011/11/exporting-versioned-configurations/"/>
    <updated>2011-11-02T17:24:33-05:00</updated>
    <id>http://getwallaby.com/2011/11/exporting-versioned-configurations</id>
    <content type="html"><![CDATA[<p>Wallaby stores versioned configurations in a database.  <a href="http://getwallaby.com/the-wallaby-api/">Wallaby API</a> clients can access older versions of a node&#8217;s configuration by supplying the <code>version</code> option to the <code>Node#getConfig</code> method.  Sometimes, though, we&#8217;d like to inspect individual configurations in greater detail than the API currently allows.</p>

<p>The <a href="http://git.fedorahosted.org/git/?p=grid/wallaby.git">Wallaby git repository</a> now contains a <a href="http://git.fedorahosted.org/git/?p=grid/wallaby.git;a=blob;f=lib/mrg/grid/config/shell/cmd_versioned_config_export.rb;hb=HEAD">command</a> to export versioned configurations from a database to flat text files.  Clone the repository or just download the file and then place <code>cmd_versioned_config_export.rb</code> somewhere in your <code>WALLABY_SHELL_COMMAND_PATH</code>, and you&#8217;ll be able to invoke it like this:</p>

<pre><code>Usage:  wallaby vc-export DBFILE
exports versioned configurations from DBFILE to plain text files
    -h, --help                       displays this message
        --verbose                    outputs information about command progress
    -o, --output-dir DIR             set output directory to DIR
        --earliest NUM               output only the earliest NUM configurations
        --latest NUM                 output only the most recent NUM configurations
        --since DATE                 output only configurations since the given date
</code></pre>

<p>It will create a directory (called <code>snapshots</code> by default) with subdirectories for each versioned configuration; each of these will be timestamped with the time of the configuration.  Within each version directory, there will be directories for node configurations and stored group configurations.  (If you&#8217;re using an older version of Wallaby, the only stored group configuration will be for the default group.  Versioned configurations generated with a fairly recent version of Wallaby, on the other hand, will have stored configurations for more groups and should also have useful information about node memberships in the configuration file.)</p>

<p><code>wallaby vc-export</code> allows you both to back up versioned configurations and simply to inspect them.  Let us know if you find a new application for it, or if you find it useful in other ways!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Wallaby paper at SC11]]></title>
    <link href="http://getwallaby.com/2011/09/wallaby-paper-at-sc11/"/>
    <updated>2011-09-08T12:07:30-05:00</updated>
    <id>http://getwallaby.com/2011/09/wallaby-paper-at-sc11</id>
    <content type="html"><![CDATA[<p>I&#8217;m pleased to announce that our paper &#8221;<a href="http://web.willbenton.com/research/sc11">Wallaby: A Scalable Semantic Configuration Service for Grids and Clouds</a>&#8221; will be <a href="http://sc11.supercomputing.org/schedule/event_detail.php?evid=sprac111">presented at SC11 in the &#8220;State of the Practice&#8221; track on Clouds and Grids</a>.  If you&#8217;re going to be at SC, I hope we&#8217;ll be able to chat about Wallaby!</p>

<p>(Cross-posted from <a href="http://chapeau.freevariable.com/">Chapeau</a>.)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Node tagging in a Wallaby client library]]></title>
    <link href="http://getwallaby.com/2011/06/node-tagging-in-a-wallaby-client-library/"/>
    <updated>2011-06-20T11:11:01-05:00</updated>
    <id>http://getwallaby.com/2011/06/node-tagging-in-a-wallaby-client-library</id>
    <content type="html"><![CDATA[<p>In an <a href="http://getwallaby.com/2011/using-wallaby-groups-to-implement-node-tagging/">earlier post</a>, I presented a technique for adding <em>node tagging</em> to Wallaby without adding explicit tagging support to the Wallaby API.  Node tags are useful for a variety of reasons:  they can correspond to informal user-supplied classifications of nodes or machine-generated system attributes (e.g. &#8220;64-bit&#8221;, &#8220;high-memory&#8221;, &#8220;infiniband&#8221;).  Since we implemented tags as special Wallaby groups, they may contain configuration information (although they don&#8217;t need to) and will be advertised as one of the machine&#8217;s Condor attributes.</p>

<p>The Wallaby source repository now includes a <a href="http://git.fedorahosted.org/git/?p=grid/wallaby.git;a=blob;f=extensions/tagging.py;hb=HEAD">small Python module</a> that patches the <a href="http://chapeau.freevariable.com/2010/12/extending-wallaby-with-a-python-client-library.html">Wallaby python client library</a> with idiomatic tagging support:  namely, you can ask the store for the name of the group that represents the partition between tags and groups, and you can call getTags() and modifyTags() methods on Node objects, as well as inspect the tags property of a Node.  This will appear in Wallaby packages in the future, but you don&#8217;t have to be that adventurous to try it out now!  To use it, clone the repository and put extensions/tagging.py and schema/wallaby.py somewhere in your PYTHONPATH.  Then follow along with this transcript:</p>

<figure class='code'><figcaption><span>tagging-example.py </span><a href='https://gist.github.com/1035680'>Fork this example on GitHub </a></figcaption> <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">import</span> <span class="nn">wallaby</span>
</span><span class='line'>
</span><span class='line'><span class="c"># the &quot;tagging&quot; module patches the Wallaby client library with </span>
</span><span class='line'><span class="c"># support for tag operations</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">tagging</span>
</span><span class='line'>
</span><span class='line'><span class="c"># We&#39;ll start by setting up a Wallaby client library session against</span>
</span><span class='line'><span class="c"># the broker on localhost</span>
</span><span class='line'><span class="kn">from</span> <span class="nn">qmf.console</span> <span class="kn">import</span> <span class="n">Session</span>
</span><span class='line'><span class="n">console</span> <span class="o">=</span> <span class="n">Session</span><span class="p">()</span>
</span><span class='line'><span class="n">console</span><span class="o">.</span><span class="n">addBroker</span><span class="p">()</span>
</span><span class='line'><span class="n">raw_store</span><span class="p">,</span> <span class="o">=</span> <span class="n">console</span><span class="o">.</span><span class="n">getObjects</span><span class="p">(</span><span class="n">_class</span><span class="o">=</span><span class="s">&quot;Store&quot;</span><span class="p">)</span>
</span><span class='line'><span class="n">store</span> <span class="o">=</span> <span class="n">wallaby</span><span class="o">.</span><span class="n">Store</span><span class="p">(</span><span class="n">raw_store</span><span class="p">,</span> <span class="n">console</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c"># call this method after the store client is initialized so that</span>
</span><span class='line'><span class="c"># the tagging library knows how to create missing groups</span>
</span><span class='line'><span class="n">tagging</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="n">store</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c"># Clear out all of the memberships on a fake node</span>
</span><span class='line'><span class="n">store</span><span class="o">.</span><span class="n">getNode</span><span class="p">(</span><span class="s">&quot;fake-node.example.com&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">modifyMemberships</span><span class="p">(</span><span class="s">&quot;REPLACE&quot;</span><span class="p">,</span> <span class="p">[],</span> <span class="p">{})</span>
</span><span class='line'>
</span><span class='line'><span class="c"># After clearing a node&#39;s membership list, it will have no tags</span>
</span><span class='line'><span class="n">store</span><span class="o">.</span><span class="n">getNode</span><span class="p">(</span><span class="s">&quot;fake-node.example.com&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getTags</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="c"># You can also access the tags of a given node via its &quot;tags&quot; attribute</span>
</span><span class='line'><span class="n">store</span><span class="o">.</span><span class="n">getNode</span><span class="p">(</span><span class="s">&quot;fake-node.example.com&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">tags</span>
</span><span class='line'>
</span><span class='line'><span class="c"># by convention, we&#39;re preceding tags with &quot;@&quot; here</span>
</span><span class='line'><span class="n">store</span><span class="o">.</span><span class="n">getNode</span><span class="p">(</span><span class="s">&quot;fake-node.example.com&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">modifyTags</span><span class="p">(</span><span class="s">&quot;ADD&quot;</span><span class="p">,</span> <span class="p">[</span><span class="s">&quot;@Foo&quot;</span><span class="p">,</span> <span class="s">&quot;@Blitz&quot;</span><span class="p">],</span> <span class="n">create_missing_tags</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c"># This should return [&quot;@Foo&quot;, &quot;@Blitz&quot;]</span>
</span><span class='line'><span class="n">store</span><span class="o">.</span><span class="n">getNode</span><span class="p">(</span><span class="s">&quot;fake-node.example.com&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">tags</span>
</span><span class='line'>
</span><span class='line'><span class="c"># This should return [&quot;===TAGS_BELOW===&quot;, &quot;@Foo&quot;, &quot;@Blitz&quot;]</span>
</span><span class='line'><span class="n">store</span><span class="o">.</span><span class="n">getNode</span><span class="p">(</span><span class="s">&quot;fake-node.example.com&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">memberships</span>
</span></code></pre></td></tr></table></div></figure>


<p>I appreciate any feedback or comments on this library.</p>

<p>(Cross-posted from <a href="http://chapeau.freevariable.com/">Chapeau</a>.)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Using Wallaby groups to implement node tagging]]></title>
    <link href="http://getwallaby.com/2011/05/using-wallaby-groups-to-implement-node-tagging/"/>
    <updated>2011-05-25T10:03:01-05:00</updated>
    <id>http://getwallaby.com/2011/05/using-wallaby-groups-to-implement-node-tagging</id>
    <content type="html"><![CDATA[<p>One of the great things about Wallaby is that it&#8217;s a platform, not merely a tool.  Put another way, if it doesn&#8217;t do exactly what you want, you can use its <a href="http://getwallaby.com/the-wallaby-api/">API</a> to build tools that benefit from configuration validation and deployment.  We&#8217;ve talked in the past about a <a href="http://chapeau.freevariable.com/2010/09/migrating-legacy-condor-configurations-to-wallaby.html">number</a> of <a href="http://chapeau.freevariable.com/2010/10/wallaby-node-inventory-with-constraints.html">useful</a> <a href="http://chapeau.freevariable.com/2010/10/retrieving-wallaby-node-configurations-over-http.html">tools</a> built on the Wallaby API.  (Another cool Wallaby API client that I hope to talk about more in the future is <a href="https://github.com/erikerlandson/">Erik Erlandson</a>&#8217;s <a href="http://git.fedorahosted.org/git/?p=grid/albatross.git">Albatross</a> project, which programmatically generates and changes pool configurations in order to test Condor scale and functionality.)</p>

<p>The Wallaby API is designed to be sufficiently general to allow developers to do just about anything with configuration data, not to unnecessarily restrict users to a few use cases that we thought of.  Because of this generality, some tasks might require adopting application-level conventions.  In this article, we&#8217;ll cover one such convention and see how the Wallaby API is flexible enough to handle an interesting use case &#8211; namely, tagging nodes with various keywords, perhaps as supplied by a user or generated by an agent like <a href="http://qpidcomponents.org/download.html">sesame</a> or <a href="https://github.com/matahari/matahari/wiki">matahari</a>.</p>

<p>First, though, we&#8217;ll review how configurations are generated and applied to nodes.  Recall that a <em>node</em> is a member of several <em>groups</em>.  These memberships are ordered:  a node&#8217;s lowest-priority membership is always in the special <em>default group</em> (which includes every node), and its highest-priority membership is always in a special, node-specific <em>identity group</em> (which only includes a single node).  Zero or more memberships in <em>explicit groups</em> may occupy the priority space between the default group and a node&#8217;s identity group.  In the illustration below, <code>node.local</code> is a member of two explicit groups, which have blue backgrounds:  &#8220;EC2 submit nodes,&#8221; and &#8220;Execute nodes.&#8221;</p>

<p><img src="http://chapeau.freevariable.com//wallaby-groups-simple.png" alt="Wallaby groups simple" /></p>

<p>When Wallaby calculates the configuration for <code>node.local.</code>, it will begin with a copy of the default group&#8217;s configuration.  It will then repeatedly apply the configurations for the explicit groups and identity group in order of increasing priority, so that parameter-value mappings from higher-priority groups take precedence over lower-priority ones (and thus either replace these or are appended to them, depending on the mapping kind).  The <code>condor_configd</code>, which takes a node&#8217;s configuration from Wallaby and installs it on a node, will also cause the Wallaby groups a node is a member of (as well as the Wallaby features it has installed) to be advertised for use in Condor matchmaking.  So a Condor job could specify that it wanted to match against a node that was a member of the &#8220;EC2 submit nodes&#8221; group; this would translate into a preference for <code>node.local</code>.</p>

<p>Because group names appear in machine ClassAd attributes, the list of explicit node memberships is a natural place to put tag information.  Nodes would simply be members of &#8220;dummy&#8221; groups like &#8220;Desktop workstations&#8221; or &#8220;Machines with more than 8GB RAM,&#8221; and these keywords could be used in matchmaking or in searching for particular nodes.  However, the list of explicit groups is not necessarily suitable for automatic manipulation:  users will not necessarily expect their changes to be overridden, other API clients won&#8217;t necessarily expect users to rearrange or remove groups corresponding to tags, and in general it is impossible to determine whether a group membership should be interpreted as a tag or as an explicit membership.</p>

<p>We can adopt a convention to partition the space of group memberships.  Say we create a special <em>sentinel group</em> to partition the membership space:  every node will be a member of the sentinel group.  All memberships that are of a lower priority than the sentinel group will be managed by tagging agents, and all memberships that are of a higher priority will be explicitly managed by the user.  In the example below, &#8220;&#8211;EXPLICIT GROUPS&#8221; is the sentinel group, and groups in yellow correspond to tags.</p>

<p><img src="http://chapeau.freevariable.com//wallaby-sentinel-groups.png" alt="Wallaby memberships, with a sentinel group and tag groups" /></p>

<p>This approach demonstrates the generality of the Wallaby API, and allows users to supply tag-specific configurations by installing parameters or features on tag groups, for example, ensuring that every desktop workstation has a policy that favors its owner, or overprovisioning high-memory nodes.  However, it is not free of shortcomings.  Firstly, Wallaby will present all of these groups &#8211; sentinel, tag, and explicit groups &#8211; so a user-facing tool that puts a friendly interface on node memberships and tagging will need to use the sentinel group to partition which groups it displays to the user.  Secondly, the sentinel groups must be added to all nodes (in practice, we can assume that the absence of a sentinel group implies the absence of tags); in general, we are relying on clients to enforce invariants in this case.  Finally, altering a node&#8217;s membership list to insert a tag will cause that node&#8217;s configuration to be validated at the next activation.  If the node has a complex configuration, this could be expensive.</p>

<p>(Cross-posted from <a href="http://chapeau.freevariable.com/">Chapeau</a>.)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Wallaby user tutorial and live VM]]></title>
    <link href="http://getwallaby.com/2011/05/wallaby-user-tutorial-and-live-vm/"/>
    <updated>2011-05-04T13:03:01-05:00</updated>
    <id>http://getwallaby.com/2011/05/wallaby-user-tutorial-and-live-vm</id>
    <content type="html"><![CDATA[<p>Rob Rati and I presented a Wallaby user tutorial at <a href="http://www.cs.wisc.edu/condor/CondorWeek2011/">Condor Week</a> yesterday.  Today, we have a <a href="http://getwallaby.com/tutorial-2011/">tutorial you can follow along with at home</a>, including a link to an EC2 AMI that boots up with Wallaby and Condor installed and running so you can experiment without installing anything. Enjoy!</p>

<p>(Cross-posted from <a href="http://chapeau.freevariable.com/">Chapeau</a>.)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Extending Wallaby with a Python client library]]></title>
    <link href="http://getwallaby.com/2010/12/extending-wallaby-with-a-python-client-library/"/>
    <updated>2010-12-10T09:57:01-06:00</updated>
    <id>http://getwallaby.com/2010/12/extending-wallaby-with-a-python-client-library</id>
    <content type="html"><![CDATA[<p>In <a href="http://chapeau.freevariable.com/2010/10/extending-the-wallaby-shell.html">my previous post</a>, we saw how to extend wallaby by writing Ruby classes that use a client library to extend the wallaby shell.  If you&#8217;re comfortable with Ruby, this is a great way to build functionality on top of the wallaby API in an idiomatic way.  However, Python programmers shouldn&#8217;t have to learn Ruby just to interact with wallaby.</p>

<p>The wallaby source repository now includes <a href="http://git.fedorahosted.org/git/?p=grid/wallaby.git;a=blob;f=schema/wallaby.py">a Python client library</a> to interact with a wallaby service.  This library does a few things to make your life easier:</p>

<ol>
<li>Raw QMF object proxies are wrapped in Python client objects.</li>
<li>Object references returned by QMF methods are automatically wrapped by client objects.</li>
<li>QMF methods that return error statuses will raise exceptions in the client library.</li>
<li>Client classes include docstrings for QMF methods.</li>
</ol>


<p>See below for a transcript illustrating these features.  Note that you&#8217;ll need to have the following packages installed to use the Python wallaby client library:  Red Hat Enterprise MRG 1.3 (or the <code>python-qmf</code> package from a recent Fedora release) and Wallaby 0.9.18 or later (0.10.0 to access all of the features exposed through the client library).</p>

<figure class='code'><figcaption><span>Example interaction with the Wallaby Python client library.   </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">qmf.console</span> <span class="kn">import</span> <span class="n">Session</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">wallaby</span>
</span><span class='line'>
</span><span class='line'><span class="c"># create a new console object</span>
</span><span class='line'><span class="n">console</span> <span class="o">=</span> <span class="n">Session</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="c"># connect to the broker (on localhost:5672, by default)</span>
</span><span class='line'><span class="n">console</span><span class="o">.</span><span class="n">addBroker</span><span class="p">()</span>
</span><span class='line'><span class="c"># =&gt; Broker connected at: localhost:5672</span>
</span><span class='line'>
</span><span class='line'><span class="c"># find the QMF object for the wallaby service</span>
</span><span class='line'><span class="n">raw_store</span><span class="p">,</span> <span class="o">=</span> <span class="n">console</span><span class="o">.</span><span class="n">getObjects</span><span class="p">(</span><span class="n">_class</span><span class="o">=</span><span class="s">&quot;Store&quot;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c"># wrap it up in a client object</span>
</span><span class='line'><span class="n">store</span> <span class="o">=</span> <span class="n">wallaby</span><span class="o">.</span><span class="n">Store</span><span class="p">(</span><span class="n">raw_store</span><span class="p">,</span> <span class="n">console</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c"># now, interact with it!</span>
</span><span class='line'><span class="n">node</span> <span class="o">=</span> <span class="n">store</span><span class="o">.</span><span class="n">addNode</span><span class="p">(</span><span class="s">&quot;barney.local.&quot;</span><span class="p">)</span>
</span><span class='line'><span class="n">feature</span> <span class="o">=</span> <span class="n">store</span><span class="o">.</span><span class="n">addFeature</span><span class="p">(</span><span class="s">&quot;Example feature&quot;</span><span class="p">)</span>
</span><span class='line'><span class="n">param</span> <span class="o">=</span> <span class="n">store</span><span class="o">.</span><span class="n">addParam</span><span class="p">(</span><span class="s">&quot;EXAMPLE_PARAM&quot;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c"># most &quot;options&quot; arguments are indeed optional</span>
</span><span class='line'><span class="n">feature</span><span class="o">.</span><span class="n">modifyParams</span><span class="p">(</span><span class="s">&quot;ADD&quot;</span><span class="p">,</span> <span class="p">{</span><span class="s">&quot;EXAMPLE_PARAM&quot;</span><span class="p">:</span><span class="s">&quot;example value&quot;</span><span class="p">})</span>
</span><span class='line'><span class="n">store</span><span class="o">.</span><span class="n">getDefaultGroup</span><span class="p">()</span><span class="o">.</span><span class="n">modifyFeatures</span><span class="p">(</span><span class="s">&quot;ADD&quot;</span><span class="p">,</span> <span class="p">[</span><span class="s">&quot;Example feature&quot;</span><span class="p">])</span>
</span><span class='line'>
</span><span class='line'><span class="n">node</span><span class="o">.</span><span class="n">getConfig</span><span class="p">()</span>
</span><span class='line'><span class="c"># =&gt; {u&#39;WALLABY_CONFIG_VERSION&#39;: u&#39;0&#39;, u&#39;EXAMPLE_PARAM&#39;: u&#39;example value&#39;}</span>
</span><span class='line'>
</span><span class='line'><span class="c"># What does the activateConfiguration method return?</span>
</span><span class='line'><span class="n">help</span><span class="p">(</span><span class="n">store</span><span class="o">.</span><span class="n">activateConfiguration</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c"># Help on method activateConfiguration in module wallaby:</span>
</span><span class='line'><span class="c"># </span>
</span><span class='line'><span class="c"># activateConfiguration(self) method of wallaby.Store instance</span>
</span><span class='line'><span class="c">#     Returns a tuple consisting of:</span>
</span><span class='line'><span class="c">#     * A map containing an explanation of why the configuration isn&#39;t valid, or an empty map if the configuration was successfully activated.</span>
</span><span class='line'><span class="c">#     * A set of warnings encountered during configuration activation.</span>
</span><span class='line'>
</span><span class='line'><span class="n">store</span><span class="o">.</span><span class="n">activateConfiguration</span><span class="p">()</span>
</span><span class='line'><span class="c"># =&gt; ({}, [])</span>
</span><span class='line'>
</span><span class='line'><span class="n">node</span><span class="o">.</span><span class="n">getConfig</span><span class="p">()</span>
</span><span class='line'><span class="c"># =&gt; {u&#39;WALLABY_CONFIG_VERSION&#39;: u&#39;1292513031757418&#39;, u&#39;EXAMPLE_PARAM&#39;: u&#39;example value&#39;}</span>
</span><span class='line'>
</span><span class='line'><span class="c"># try something that will cause an error</span>
</span><span class='line'>
</span><span class='line'><span class="n">feature</span> <span class="o">=</span> <span class="n">store</span><span class="o">.</span><span class="n">addFeature</span><span class="p">(</span><span class="s">&quot;Example feature&quot;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c"># Traceback (most recent call last):</span>
</span><span class='line'><span class="c">#   File &quot;&lt;stdin&gt;&quot;, line 2, in &lt;module&gt;</span>
</span><span class='line'><span class="c">#   File &quot;wallaby.py&quot;, line 404, in addFeature</span>
</span><span class='line'><span class="c">#     raise ClientError(result.status, result.text)</span>
</span><span class='line'><span class="c"># wallaby.ClientError: (67109121, u&#39;Feature name Example feature is already taken&#39;)</span>
</span></code></pre></td></tr></table></div></figure>


<p>(Cross-posted from <a href="http://chapeau.freevariable.com/">Chapeau</a>.)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Extending the Wallaby shell]]></title>
    <link href="http://getwallaby.com/2010/10/extending-the-wallaby-shell/"/>
    <updated>2010-10-21T11:43:07-05:00</updated>
    <id>http://getwallaby.com/2010/10/extending-the-wallaby-shell</id>
    <content type="html"><![CDATA[<p>The most recent few releases of the Wallaby configuration management service have included some great new features:  <code>wallaby console</code> can now be used as an interpreter for <a href="http://en.wikipedia.org/wiki/Shebang_(Unix">shebang scripts</a>), <code>wallaby inventory</code> now supports limiting the nodes it lists with <a href="http://chapeau.freevariable.com/2010/10/wallaby-node-inventory-with-constraints.html">expressive constraints</a>, the <code>wallaby http-server</code> command <a href="http://chapeau.freevariable.com/2010/10/retrieving-wallaby-node-configurations-over-http.html">provides access to node configurations over HTTP</a>, and the <code>wallaby</code> command and its subcommands feature many minor functional and aesthetic improvements.  One feature that I&#8217;m particularly excited about is under the hood:  there is now an API for extending the wallaby shell with your own subcommands.  In this post, we&#8217;ll look at how to make a new <code>wallaby</code> subcommand.  <noscript>You&#8217;ll need to view this post outside of a feed reader, in a browser with JavaScript support, in order to see the embedded code snippets.</noscript></p>

<p>First, make sure you&#8217;re running <a href="http://git.fedorahosted.org/git/?p=grid/wallaby.git;a=shortlog;h=refs/tags/v0.9.23">wallaby 0.9.23</a> or later.  If you&#8217;re running Fedora, you can simply <code>git clone</code> the repository, check out the <code>v0.9.23</code> tag, and run <code>rake rpms</code> to generate installable packages.</p>

<p>Next, you&#8217;ll want to have a directory in which to put your custom wallaby shell commands.  I recommend putting them somewhere in your home directory.  (For the purposes of this discussion, we&#8217;ll assume that you&#8217;re using <code>~/.wallaby</code>.)  The <code>wallaby</code> command will find and attempt to load every ruby file in this directory that begins with <code>cmd_</code>, so you probably don&#8217;t want it to be writable by anyone other than you.  Once you have this directory set up, set <code>WALLABY_COMMAND_DIR</code> in your environment to this directory:</p>

<p><code>export WALLABY_COMMAND_DIR=${HOME}/.wallaby</code></p>

<p>We&#8217;ll start by making a fairly straightforward command that simply prints out the API version supported by the wallaby service.  To create a new wallaby command, use <code>wallaby new-command</code>:</p>

<script src="http://gist.github.com/638780.js?file=gistfile3.txt"></script>


<p>The <code>-d</code> option allows us to provide a description for the new command (viz., what would show up if we typed <code>wallaby help commands</code>).  The <code>-D</code> option allows us to specify a directory in which to create the new command file.  (<code>wallaby new-command</code> supports several other options; you can use <code>wallaby help new-command</code> for additional documentation.)  After executing this command, we&#8217;ll have a documented template file for a new wallaby command, called <code>cmd_api_version.rb</code>, installed in our <code>WALLABY_COMMAND_DIR</code>.  It will look something like this:</p>

<script src="http://gist.github.com/638780.js?file=generated_cmd_api_version.rb"></script>


<p>We can now run <code>wallaby help commands</code> and verify that our command shows up.  Sure enough, it does:</p>

<script src="http://gist.github.com/638780.js?file=gistfile4.txt"></script>


<p>We can even run <code>wallaby api-version</code>, although it won&#8217;t do anything yet.  To make it do something useful, we&#8217;re going to edit the <code>act</code> method defined for us in the <code>ApiVersion</code> class.  We could do something very simple, like insert <code>puts store.apiVersionNumber</code> before <code>return 0</code>, but it would be nice to allow a user to format the API version number as he or she sees fit, in order to use the result of our command in other scripts.</p>

<p>To let the user supply a format string, we&#8217;ll need to add a command-line option, which requires us to edit the <code>init_option_parser</code> method.  This method must return an <code>OptionParser</code> object; if you haven&#8217;t used Ruby&#8217;s option parser class, read up on <a href="http://ruby-doc.org/stdlib/libdoc/optparse/rdoc/classes/OptionParser.html">its documentation</a>.  For this example, though, the changes we have to make are pretty minor.</p>

<p>After we&#8217;ve added a command-line option and a body for the <code>act</code> method, our <code>cmd_api_version.rb</code> file will look more like this:</p>

<script src="http://gist.github.com/638780.js?file=cmd_api_version.rb"></script>


<p>Running this command without arguments will return a string like <code>The wallaby service is running version 20100915 of the Wallaby API.</code>  If you supply a format string, you can get simply the raw number (with <code>--format '%s'</code>) or something more suited to your taste.  (Note that this example script doesn&#8217;t do any sanity- or error-checking of the format string, which you&#8217;d certainly want to do if you were going to put your command into production.)</p>

<p>We&#8217;ll look at a more interesting example wallaby shell command and some of the issues that more interesting commands raise in a future post.</p>

<p>(Cross-posted from <a href="http://chapeau.freevariable.com/">Chapeau</a>.)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Wallaby node inventory with constraints]]></title>
    <link href="http://getwallaby.com/2010/10/wallaby-node-inventory-with-constraints/"/>
    <updated>2010-10-18T11:59:37-05:00</updated>
    <id>http://getwallaby.com/2010/10/wallaby-node-inventory-with-constraints</id>
    <content type="html"><![CDATA[<p>wallaby inventory is a useful command for quickly checking up on the health of your pool and answering certain kinds of questions:  Which nodes have checked in recently?  Which nodes have checked in without being explicitly configured?  Which nodes have never checked in?</p>

<p>Recently, I added <em>constraint</em> support to wallaby inventory.  That is, instead of listing all nodes, you can choose to list only nodes for which a given Ruby expression is true.  These constraints can be simple, like last_checkin &lt; 1.hour_ago (which will find all nodes that have checked in more than one hour ago), or more complex.  They are only limited by Ruby&#8217;s <a href="http://ruby-doc.org/docs/ProgrammingRuby/html/taint.html">$SAFE execution</a>:  among other things, they cannot modify running classes, perform file I/O, fork new processes, or, because of how QMF is implemented, invoke QMF <em>methods</em> on the node object.  However, every QMF <em>property</em> of the node object is available to wallaby inventory constraints.  The transcript below provides some examples running against a wallaby agent loaded up with test data.</p>

<script src="http://gist.github.com/632556.js"> </script>


<p><noscript>Visit this post in a browser with JavaScript support (not a feed reader) to see the embedded transcript.</noscript></p>

<p>(Cross-posted from <a href="http://chapeau.freevariable.com/">Chapeau</a>.)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Retrieving Wallaby configurations over HTTP]]></title>
    <link href="http://getwallaby.com/2010/10/retrieving-wallaby-configurations-over-http/"/>
    <updated>2010-10-15T13:01:54-05:00</updated>
    <id>http://getwallaby.com/2010/10/retrieving-wallaby-configurations-over-http</id>
    <content type="html"><![CDATA[<p>In some environments, users may wish to use Wallaby to serve configurations to nodes that can&#8217;t reach the Qpid broker that the Wallaby agent is running against.  Some users may also want to write simple scripts that access current or historical configuration data without writing a QMF console.  The wallaby http-server command, which is available beginning with wallaby-0.9.19, offers a solution for such users.  It provides a read-only web service gateway to a node&#8217;s configuration:  the last activated configuration, a particular version, and the node&#8217;s current configuration in the store (which may not yet have been activated).</p>

<p>wallaby http-server requires that the Sinatra framework is installed.  If you are building RPMs for wallaby, wallaby-http-server will be built as a separate package, and only if you&#8217;re building from Fedora 12 or later.  If you&#8217;re installing from source, simply gem install sinatra before running the HTTP server; if you&#8217;ve already installed a wallaby package on a non-Fedora system, you can simply install cmd_http_server.rb <a href="http://git.fedorahosted.org/git/?p=grid/wallaby.git;a=blob;f=lib/mrg/grid/config/shell/cmd_http_server.rb;hb=HEAD">from the source repository</a> into your wallaby shell commands directory.  (In the future, we expect to package wallaby-http-server for Red Hat Enterprise Linux as well, thus simplifying this process.)</p>

<p>The transcript below shows the HTTP server&#8217;s built-in help, listing all of the methods it supports (as of wallaby-0.9.20, which will be released later today).  Please let us know if you can think of other API methods that would be useful to expose over the Wallaby HTTP server.</p>

<script src="http://gist.github.com/628571.js"> </script>


<p><noscript>Please visit this post in a browser with JavaScript support to see the embedded code snippet.</noscript></p>

<p><em>(Cross-posted from <a href="http://chapeau.freevariable.com/2010/10/retrieving-wallaby-node-configurations-over-http.html">chapeau</a>.)</em></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Flexible interaction with the Wallaby console]]></title>
    <link href="http://getwallaby.com/2010/10/flexible-interaction-with-the-wallaby-console/"/>
    <updated>2010-10-06T09:56:35-05:00</updated>
    <id>http://getwallaby.com/2010/10/flexible-interaction-with-the-wallaby-console</id>
    <content type="html"><![CDATA[<p>One of the main benefits of using Wallaby for configuration is the remote-access <a href="http://getwallaby.com/the-wallaby-api/">API</a>.  Because the API is comprehensive and usable from any language with a QMF binding (including C++, Java, Python, and Ruby), it provides developers and users with the means to build tools, policies, templating systems, and one-off scripts on top of Wallaby.</p>

<p>If you&#8217;re writing a Wallaby API client in Ruby, you can use a couple of nice features to ease development:  the <a href="http://git.fedorahosted.org/git/?p=grid/wallaby.git;a=blob;f=lib/mrg/grid/config-client.rb;hb=HEAD">Wallaby client library</a>, which presents a more polished interface than dealing with raw QMF queries and calls, and the <code>wallaby console</code> utility, which connects to a specified broker and gives you a Ruby <a href="http://en.wikipedia.org/wiki/Read-eval-print_loop">REPL</a> that has a global variable pointing to a store client &#8212; perfect for casual, interactive experimentation.</p>

<p>Use the same global command line options for <code>wallaby console</code> as you&#8217;d use for any other <code>wallaby</code> subcommand:  specifying host, port, and authentication information for the broker.  Once you&#8217;ve started it up, you&#8217;ll be dropped into a Ruby prompt where <code>Wallaby::store</code> is a reference to a client proxy object for your Wallaby store.  You can then inspect or modify any entity that the store knows about; the store object has accessors called <code>nodes</code>, <code>features</code>, <code>parameters</code>, etc., that return arrays of each kind of object.  Then you can invoke methods on each object.  The transcript below shows a session in which the user inspects some details of a node in a trivial pool:</p>

<pre><code>&gt;&gt; Wallaby::store.nodes.map {|n| n.name}
=&gt; ["frotz"]
&gt;&gt; frotz = Wallaby::store.nodes[0] ; nil
=&gt; nil
&gt;&gt; Wallaby::store.groups.map {|g| g.name}
=&gt; ["+++DEFAULT", "+++af413ebf1de0f9d4253eb89717a4e13b"]
&gt;&gt; Wallaby::store.groups[1].features
=&gt; ["DisablePreemption"]
&gt;&gt; frotz.getConfig
=&gt; {"PREEMPT"=&gt;"FALSE", "WALLABY_CONFIG_VERSION"=&gt;"1284180448517726"}
</code></pre>

<p>In the future, it will be easier to extend the <code>wallaby</code> command with your own subcommands.  For now, however, <code>wallaby console</code> provides a great way to interact with the Wallaby API.</p>

<p><em>(Cross-posted from <a href="http://chapeau.freevariable.com/2010/09/flexible-interaction-with-the-wallaby-console.html">Chapeau</a>.)</em></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Migrating legacy Condor configurations to Wallaby]]></title>
    <link href="http://getwallaby.com/2010/10/migrating-legacy-condor-configurations-to-wallaby/"/>
    <updated>2010-10-06T09:32:01-05:00</updated>
    <id>http://getwallaby.com/2010/10/migrating-legacy-condor-configurations-to-wallaby</id>
    <content type="html"><![CDATA[<p><a href="http://getwallaby.com">Wallaby</a> provides a great way to manage Condor configurations, and if you&#8217;re just starting out with Condor, it&#8217;s easy to do things the Wallaby way from the start.  However, many installations have their own systems for managing Condor features and configuration files &#8212; perhaps with separate files representing common, node-specific, and feature-specific configurations.  In this post, we&#8217;ll talk about wallaby feature-import, a command that lets you import Condor configuration-file snippets into the Wallaby database as Wallaby features, complete with metadata and relationships with other features.</p>

<p>The basic syntax of the file you&#8217;ll pass to wallaby feature-import is the same as the basic syntax of a Condor configuration file; in fact, you can import many Condor configuration files without modification.  (Only the NAME = VALUE syntax is supported at this time; you can&#8217;t use the NAME = +VALUE syntax for DC_DAEMON_LIST.)  However, there are some special directives that you can pass to the configuration file as comments:</p>

<pre><code>#name NAME
</code></pre>

<p>Indicates that this feature should be named <em>NAME</em>.  (Note that you must supply a name for the feature, either on the command line, via -n NAME, or in the file itself.)</p>

<pre><code>#includes FEATURE
</code></pre>

<p>Indicates that this feature should include <em>FEATURE</em>, which must exist in Wallaby.</p>

<pre><code>#dependson FEATURE
</code></pre>

<p>Indicates that this feature should depend upon <em>FEATURE</em>, which must exist in Wallaby.</p>

<pre><code>#default PARAM
</code></pre>

<p>Indicates that this feature should include <em>PARAM</em>, and set it to its default value.</p>

<p>Because Condor ignores these comments, it is possible to use the same file both as a Condor configuration file and as a Wallaby feature description.  (Other comments will be ignored by wallaby feature-import, but they may generate harmless warnings.)  wallaby feature-import will automatically create any parameters referenced in the file that don&#8217;t already exist in Wallaby&#8217;s database; this can be a great time-saver for migrating to Wallaby from a custom configuration-management system.</p>

<p>The following example file is a feature that defines eight startds on a single node:</p>

<script src="http://gist.github.com/599099.js?file=EightStartds.txt"></script>


<p><noscript>Please visit this post in a browser with JavaScript support to see the embedded code sample.</noscript></p>

<p><em>(Cross-posted from <a href="http://chapeau.freevariable.com/2010/09/migrating-legacy-condor-configurations-to-wallaby.html">Chapeau</a>.)</em></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[API documentation]]></title>
    <link href="http://getwallaby.com/2010/05/api-documentation/"/>
    <updated>2010-05-13T17:47:11-05:00</updated>
    <id>http://getwallaby.com/2010/05/api-documentation</id>
    <content type="html"><![CDATA[<p>Basic documentation <a href="http://wallaby.freevariable.com/the-wallaby-api/">is now available</a> for the methods in the Wallaby API.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Introducing Wallaby]]></title>
    <link href="http://getwallaby.com/2010/04/introducing-wallaby/"/>
    <updated>2010-04-15T11:16:06-05:00</updated>
    <id>http://getwallaby.com/2010/04/introducing-wallaby</id>
    <content type="html"><![CDATA[<p>We had a great time at <a href="http://www.cs.wisc.edu/condor/CondorWeek2010/">Condor Week 2010</a> and are excited to announce Wallaby!   If you weren&#8217;t there, you can <a href="http://www.cs.wisc.edu/condor/CondorWeek2010/condor-presentations/benton-wallaby.pdf">check out slides from our talk</a>.  Whether you were there or not, please subscribe to <a href="http://feeds.feedburner.com/EnterTheMarsupial">our syndication feed</a> to keep up to date with new Wallaby developments, including prerelease packages, tutorials, Wallaby API tools, and official releases.</p>
]]></content>
  </entry>
  
</feed>
