Missing TestLink API Function – getExecutionResults()

I have finally come to terms with the TestLink API and have bashed out an automated test runner written in perl (which I hope to clean up and post in the next few days, there don’t seem to be many (or any) examples of this in the wild).

After getting nightly automated testing running, I wanted a very simple script which would, for each test case in a given test plan, compare the last and the second to last result and print a warning if they were different. Long term I will have better automated reporting, but our test script library is still so new that it’s not really news that a lot of them are failing. What would be news is if a specific test case passed two nights ago but failed last night.

Continue reading

Better Copy/Paste When Editing Test Steps in TestLink

My current biggest peeve with TestLink is how hard TestLink makes it to copy text out of test steps when editing a test case. Both when viewing a case and when editing test case steps via the Test Specification interface, clicking on the text of a test step causes you to start editing the text in that step.

I find this behavior very off-putting. I constantly find myself trying to copy in these views, either to copy repetitive actions from a previous step, or to do a dry-run executing a test case. I finally got mad enough today to spend the time figuring out how to change this behavior.

Update 2012-05-08: There is a usable-with-the-patch-command patch available that incorporates all of these changes and more. See the README at the base of the github repo for more details on what’s in that patch.

There are two Smarty templates that control the layout of these views, both of which live in gui/templates/testcases/. inc_steps.tpl controls the layout when viewing a test case (but not editing steps), and tcStepEdit.tpl controls the layout when editing steps.

After applying these patches, you can copy text out of the Step Action and Expected Results columns. To edit the row as before the patch, click on the row counter (the first cell in the row).

These patches are against TestLink 1.9.3

--- inc_steps.tpl.dist  2011-08-22 21:08:21.000000000 -0500
+++ inc_steps.tpl       2011-09-22 10:08:55.000000000 -0500
@@ -53,10 +53,10 @@
                                size="{#STEP_NUMBER_SIZE#}"
                                maxlength="{#STEP_NUMBER_MAXLEN#}">
                        {include file="error_icon.tpl" field="step_number"}
-                       </span>{$step_info.step_number}
+                       </span><div {if $edit_enabled} style="cursor:pointer;" onclick="launchEditStep({$step_info.id})" {/if}>{$step_info.step_number}</div>
                </td>
-               <td {if $edit_enabled} style="cursor:pointer;" onclick="launchEditStep({$step_info.id})" {/if}>{$step_info.actions}</td>
-               <td {if $edit_enabled} style="cursor:pointer;" onclick="launchEditStep({$step_info.id})" {/if}>{$step_info.expected_results}</td>
+               <td>{$step_info.actions}</td>
+               <td>{$step_info.expected_results}</td>
                {if $session['testprojectOptions']->automationEnabled}
                <td {if $edit_enabled} style="cursor:pointer;" onclick="launchEditStep({$step_info.id})" {/if}>{$gui->execution_types[$step_info.execution_type]}</td>
                {/if}
--- tcStepEdit.tpl.dist 2011-09-22 10:27:23.000000000 -0500
+++ tcStepEdit.tpl      2011-09-22 10:39:55.000000000 -0500
@@ -211,8 +211,8 @@
        {/if}
       {else}
         <td style="text-align:left;"><a href="{$hrefEditStep}{$step_info.id}">{$step_info.step_number}</a></td>
-               <td ><a href="{$hrefEditStep}{$step_info.id}">{$step_info.actions}</a></td>
-               <td ><a href="{$hrefEditStep}{$step_info.id}">{$step_info.expected_results}</a></td>
+               <td >{$step_info.actions}</td>
+               <td >{$step_info.expected_results}</td>
         {if $session['testprojectOptions']->automationEnabled}
                  <td><a href="{$hrefEditStep}{$step_info.id}">{$gui->execution_types[$step_info.execution_type]}</a></td>
                {/if}

I’ve been writing test cases with this change in place all morning and it is much, much nicer than the default behavior

Using Wiki Markup in Trac Custom Ticket Fields

Yesterday I worked on streamlining our ticket workflow in Trac. While I was in the test project anyway, I decided to address an issue that had bugged me since I first set up Trac. I originally set up two custom ticket fields related to case IDs in our CRM system:

crmnum = text
crmnum.label = Case #
crmnum.format = plain
crmnum.order = 1
crmlink = text
crmlink.label = Case Link
crmlink.format = wiki
crmlink.order = 3

The idea was that we could have the “crmnum” field to just hold the ID, which we could act on programmatically. The “crmlink” field would be a copy/paste link of the ticket’s URL, which would allow someone looking at the ticket to click through to the CRM case easily.

The problem with this is that the link is long and ugly and completely throws off the tidiness of the ticket summary.

So, having recently figured out how to write a custom wiki macro, I decided to create one for links to our CRM system also. The idea would be that the “crmlink” field would go away and the “crmnum” field could be updated to use the wiki Macro to generate the link automatically.

I nuked the “crmlink” field and changed “crmnum” from “type = plain” to “type = wiki“. I wrote my custom CRM macro, testing it in the wiki itself until I had it behaving. I then created a new ticket, entered [[CRM(123)]] in the crmnum field, and clicked “Preview”. Instead of seeing the link it should have returned, instead I got [[CRM(...)]]. I did some testing and proved to myself that some wiki processing was taking place (ticket:1 expanded correctly) and that no macro expanded in that field, including built-in macros.

Then on a hunch, I changed crmnum to “type = textarea” instead of “type = text“. Sure enough, macros work properly in textarea fields but not text fields. In a way this makes sense since almost all built-in macros return multiple lines of text. The documentation for custom ticket fields doesn’t mention that wiki processing behavior differs between these field types. Unfortunately the textarea fields can’t easily be tricked into looking like a simple text field. They come with an editing bar, and they are assigned the same class as the description field, so you can’t easily use CSS tricks on it.

I was unsure if this was a bug or intentional, but I managed to find the original Trac ticket for adding wiki processing to custom fields, and it’s clear that it was intentional to make the wiki-processing behavior of the two field types differ. Specifically, text fields are processed with format_to_oneliner() and textareas are processed with format_to_html() (this can be seen in the change to web_ui.py associated with new feature.

After figuring out that the text/textarea parsing differences were done intentionally, I lost interest in patching the code itself. Instead I looked into using InterWiki links instead of macros, which are processed in text fields.

I created trac ticket 10352 about this issue, hopefully it gets addressed one way or another.

Linking to TestLink testcases from Trac addendum

Last week I posted about a custom [[TestLink()]] macro I had written for Trac. Yesterday I wrote another macro to do something similar, but in doing so I found out that I was probably doing a lot more work than I needed to.

The macro I wrote allowed me to write [[TestLink(m3-123)]] to link directly to test case 123 in the m3 project. It was about seven lines of python and not the most complex thing in the world, but nowhere near as simple as using an InterWiki prefix. I did this by adding the following line to my Trac install’s InterMapTxt wiki page:

TestCase https://devtest.example.com/linkto.php?tprojectPrefix=$1&item=testcase&id=$1-$2

This allows me to replace [[TestLink(m3-123)]] with testcase:m3:123.

There are tradeoffs of course. The macro allows full control using python. That allowed me to write a single macro that could produce links to different places in our CRM system based on the format of the single argument to the macro. Since I don’t get that flexibility with InterWiki links, I had to use two InterWiki links instead of a single macro.

On the other hand, the InterWiki links are simpler to write, look better, and are much closer to the standard Trac wiki link style that my users are already used to. Also, they have the advantage that they are expanded in ticket custom text fields and macros aren’t.

In the end I’ll keep both the macros and the InterWiki links for these cases since they aren’t mutually exclusive, but I’ll definitely turn to InterWiki links before I turn to writing a custom macro in the future.

Arduino’s map() Function and Numeric Distribution

The Arduino map() function is an interesting beast. Very technically it works exactly as its documented to work, but not the way almost every example uses it.

Here’s an example you can find in hundreds of sketches online, including the actual documentation for map():

val = map(val, 0, 1023, 0, 255);

This is a simple map, and one would expect that every four ticks on the input would map to one tick on the output (that is, {0,1,2,3} -> 0, {4,5,6,7} -> 1, etc). But that’s not what the function above actually does.

To show the issue, let’s make the output range smaller (but still an even divisor):

val = map(val, 0, 1023, 0, 15);

This should result in an even distribution, 64 input ticks per one output tick. To test this, I wrote a quick script implementing the Arduino map logic (which they were nice enough to document):

long map(long x, long in_min, long in_max, long out_min, long out_max)
{
  return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min;
}

The script prints a table of output values and the number of times that value was returned. Here’s the output for map(?, 0, 1023, 0, 15) for each value in 0..1023:

map(0..1023, 0, 1023, 0, 15);
  0   69
  1   68
  2   68
  3   68
  4   68
  5   69
  6   68
  7   68
  8   68
  9   68
 10   69
 11   68
 12   68
 13   68
 14   68
 15    1

That’s definitely not an even distribution. Now here’s a really stark example:

map(0..1023, 0, 1023, 0, 1);
  0 1023
  1    1

That’s a pretty egregious imbalance.

I mentioned earlier that the function’s actually working how it’s documented to work, just not how it’s usually used in examples. The map() docs state that “[t]he map() function uses integer math so will not generate fractions, when the math might indicate that it should do so. Fractional remainders are truncated, and are not rounded or averaged.”

This completely makes sense – if you imagine a range of 1024 values between 0 and one, all of them will be less than 1 except the last value, and since it’s integer arithmetic, all the less-than-1 values are 0.

The solution is fairly simple – increase the in_max and out_max args by one more than the actual maximum value (and then wrap the output in constrain(), which you ought to have done anyway). It’s fairly easy to work through why this works in your head, but here are the same examples I gave above with the increased maximums:

map(0..1023, 0, 1024, 0, 2);
  0   512
  1   512
map(0..1023, 0, 1024, 0, 16);
  0   64
  1   64
  2   64
  3   64
  4   64
  5   64
  6   64
  7   64
  8   64
  9   64
 10   64
 11   64
 12   64
 13   64
 14   64
 15   64

I have worked through this and I now understand how to get the values I want out of constrain. What I don’t know, however, is why the docs and the examples don’t address this issue. I know I’m not the first person to find this issue because I’ve found sketches on the internet where people are doing this (and they’re doing it in a way that leads me to believe they’re doing it on purpose), but I couldn’t get Google to show me any pages where someone actually addresses this point. I wonder why?

As for why I was looking into this, I was playing around with a couple of pots and an LCD, making a fakey etch-a-sketch using the 2-row LCD I have for such things. I could not figure out why my vertical control would only drop me down a line when I had the pot turned 100%. Now I know…

Edit 2011-09-13:

I posted this in the arduino.cc forum last night. As I expected, this is an issue known in the arduino community. Among other things, I was referenced to this post discussing the issue. My problem with all of this isn’t so much that it’s odd behavior, but that it’s been odd for at least 2 years and there’s no mention of the oddness in the map() documentation. The docs are a wiki, so I proposed an addition in the forum thread, maybe I’ll get to add it and help someone out in the future.

Custom TestLink Macro for Trac

I’ve been spending a lot of time lately trying to get the workflow between Trac and TestLink to make sense. I’m far from having the perfect solution, but a great first step was creating a Trac wiki macro that will link to specific test cases in TestLink. There’s lots of areas in which this could be improved in the future…

Create this as conf/TestLinkMacro.py in your project (replacing test.example.com with your TestLink host):

import re

from trac.wiki.macros import WikiMacroBase

class TestLinkMacro(WikiMacroBase):
    """Macro to generate automatic links to test cases in TestLink.
    """

    def expand_macro(self, formatter, name, args):
        """Expand a TestLink test case ID into a link
        """
        TestLinkServer = 'test.example.com'
        matches = re.match(r'(.*)-(d+)$', args)
        
        return '<a href="https://' + TestLinkServer + '/linkto.php?tprojectPrefix=' + matches.group(1) + '&item=testcase&id=' + matches.group(0) + '">[[TestLink(' + matches.group(0) + ')]]</a>'

After putting this in place, and possibly restarting Trac, you can create a direct link to test case m3-123 in a wiki by typing [[TestLink(m3-123)]]. Note that I could have changed the actual string displayed in the wiki to anything I wanted, but I chose to leave it looking identical to the actual macro (but linked) so that it would reinforce to people the correct format for the test case links.

This macro is useful in several ways, but the most obvious is allowing us to enter regression test case IDs into commit messages (which then get associated with a ticket in Trac and are displayed with the wiki engine, making the test case link clickable).
With commit messages being associated with tickets automatically and being displayed by the wiki engine, this gives us a nice syntax to enter regression case links into commit messages and be very followable later

Edit 2011-09-13:

See also this update to my use of macros for linking to test cases and this comment on custom ticket field wiki parsing, both of which have a direct relation to this post

Custom Filename Completion in Term::ReadLine::Perl

When we released the last version of our product, I got an IM from a support guy saying that we had broken filename completion in the installer. My reaction was “but we don’t implement filename completion!” I ran the installer on the previous release, and sure enough, it supported filename completion. Wha?

Turns out we were getting filename completion as a side effect of using Term::ReadLine::Perl to do prompting. Support “knew” we supported it, and even had a list of known bugs they passed around, but no one in dev knew about the feature or the bug. Good times!

Anyway, fixing what we broke on completions, I started looking at the bugs support had been informally floating around. Essentially there two issues, which caused three separate problems:

1) After every completion, Term::ReadLine::Perl inserted a space character after the path
Problem 1: When auto-completing paths, this meant you had to hit delete after each path completion before you could type the next few characters.
Problem 2: If you didn’t manually delete the space from the end of the completed filename, our installer would treat the space as part of the filename and tell you the file didn’t exist.

2) By default, after every completion, Term::ReadLine::Perl “decorates” the completed file
Problem 3: If the file you were completing to was a symlink, it would have ‘@’ appended to it, and our installer would tell you the file didn’t exist.

Problems 2 & 3 were fixable with some logic after the user hit enter, and problem 1 wasn’t that annoying, but I went looking to see if I could fix the actual completion function in Term::ReadLine::Perl. I did figure it out, but since there seems to be little google-able documentation specific to addressing this in Term::ReadLine::Perl (as opposed to Term::ReadLine::Gnu) I thought I’d document what I did.

Here’s my toy app demonstrating the above issues:

$| = 1;
use Term::ReadLine;
my $Term = new Term::ReadLine 'Installer';

my $file = $Term->readline('Enter a file path: ');
print "$file = '$file'n";

The relevant library code is in Term/readline.pm. This appears to be a Perl5 wrapper around Perl4 code.

The default filename completion function is called rl_filename_list(). This function has a tunable to turn off the filename decoration (adding “@” to symlinks, etc) called $var_CompleteAddsuffix, set to “1” by default. I tried turning it off like this at the top of my app:

$readline::var_CompleteAddsuffix = 0;

This worked, in that it turned off decoration. Unfortunately, the adding of “/” to a completed path is part of decoration, and that’s functionality I wanted to keep. So, I guess I need to write my own filename expander. Here is my toy app with my custom filename expander routine:

$| = 1;
use Term::ReadLine;
my $Term = new Term::ReadLine 'Installer';

# tell Term::ReadLine::Perl to use my filename expander for completion
$readline::rl_completion_function = "main::rl_filename_list_lcl";

my $file = $Term->readline('Enter a file path: ');
print "$file = '$file'n";

exit;

sub rl_filename_list_lcl {
    my $pattern = $_[0];
    my @files = (<$pattern*>);
    foreach (@files) {
        if (-d $_) {
            $_ .= '/';
        }
    }
    return @files;
}

The subroutine rl_filename_list_lcl() is a slightly modified copy of readline::rl_filename_list(). I removed all the decoration except ‘/’ for dirs, and I removed the check of $var_CompleteAddsuffix before decorating (I want it to happen unconditionally).

That solved the decoration issue, but not the “adds a space after every completion” issue. This is controlled by $rl_completer_terminator_character being unconditionally set to ‘ ‘ in readline.pm’s complete_internal(). I experimented with setting it to '' unconditionally, which did fix my problem. However, I didn’t want to leave it like that unconditionally for fear of unexpected side effects. Then I came across this in the code:

##   $rl_completer_terminator_character -- what to insert to separate
##      a completed token from the rest.  Reset at beginning of
##      completion to ' ' so completion function can change it.

Perfect! I’m already defining a custom function, so all I have to do is set $readline::rl_completer_terminator_character to '' in it.

Et voila! My completed toy function with auto-complete behaving the way I want it to:

$| = 1;
use Term::ReadLine;
my $Term = new Term::ReadLine 'Installer';

# tell Term::ReadLine::Perl to use my filename expander for completion
$readline::rl_completion_function = "main::rl_filename_list_lcl";

my $file = $Term->readline('Enter a file path: ');
print "$file = '$file'n";

exit;

sub rl_filename_list_lcl {
    my $pattern = $_[0];
    my @files = (<$pattern*>);
    foreach (@files) {
        if (-d $_) {
            $_ .= '/';
        }
    }

    # The readline.pm library sets this to ' ' on every completion call, we can
    # force it to '' here but have to do it every time
    $readline::rl_completer_terminator_character = '';

    return @files;
}

Alternating Test Step Row Colors in TestLink

Working on implementing TestLink at work for better testing process flow. Very few complaints about the product as a whole – it can feel ponderous at times, but testing is a big problem, and TestLink is the best tool I’ve found for addressing them all.

The only real negative feedback I’ve received so far is regarding the display of steps from individual test cases. By default, the entire table is the same color, and it’s very difficult to visually track which row is which. It seems like such an obvious problem that I assumed I had missed a knob to tweak, but I can’t find anything about it in the docs, the forums, or the bug tracker.

I poked around a bit tonight and the solution turned out to be ridiculously (but pleasingly) simple.

Update 2012-05-08: There is a usable-with-the-patch-command patch available that incorporates all of these changes and more. See the README at the base of the github repo for more details on what’s in that patch.

First, add the following to gui/themes/default/css/custom.css:

.row_color_1 {
    background-color:   #DDDDDD;
}
.row_color_2 {
    background-color:   #CCCCCC;
}

Then, patch the following two files in gui/templates/testcases/ like so (original files were from 1.9.3):

--- inc_steps.tpl.dist  2011-08-22 21:08:21.000000000 -0500
+++ inc_steps.tpl       2011-08-22 21:13:02.000000000 -0500
@@ -45,7 +45,7 @@
        </tr>
        {* BUGID 3376 *}
        {foreach from=$steps item=step_info}
-       <tr id="step_row_{$step_info.step_number}">
+       <tr id="step_row_{$step_info.step_number}" class="{cycle values="row_color_1,row_color_2"}">
                <td style="text-align:left;">
                        <span class="order_info" style='display:none'>
                        <input type="text" name="step_set[{$step_info.id}]" id="step_set_{$step_info.id}"
--- tcStepEdit.tpl.dist 2011-09-22 10:27:23.000000000 -0500
+++ tcStepEdit.tpl      2011-09-22 10:39:55.000000000 -0500
@@ -197,7 +197,7 @@
   {* this means we have steps to display *}
   {if $gui->tcaseSteps != ''}
        {foreach from=$gui->tcaseSteps item=step_info}
-         <tr id="step_row_{$step_info.step_number}">
+         <tr id="step_row_{$step_info.step_number}" class="{cycle values="row_color_1,row_color_2"}">
       {if $step_info.step_number == $gui->step_number}
                    <td style="text-align:left;">{$gui->step_number}</td>
                  <td>{$steps}</td>

Blow away the cached .css files, and voila! Alternating background colors for test steps in TestLink.

Before: After:

I tried to post this as a hint in the TestLink forum but I honestly couldn’t figure out how to register a forum account. I did post a comment on a related bug in their issue tracker, ticket 3583

EDIT 2011-09-22: The original version of this post only included a change to inc_steps.tpl. Added the patch to tcStepsEdit.tpl to apply alternation to the rows in the test case step edit view also.

EDIT 2011-09-26: Updated to include the correct full path to custom.css

Enforcing a default owner for new tickets in Trac

We’ve been loving our Trac install at work, but we are still sanding down a few rough spots.  One particular spot that’s been bobbing up has been what happens when “new” tickets have an owner assigned.  I liked the ticket status being “new”, because it meant that I could easily look at my ticket list and see tickets that had been directly assigned to me that I hadn’t acknowledged yet by accepting them.  Someone else really disliked having to “accept” a ticket that was already assigned to him before he could move it to QA.

I actually managed to create a solution for this using the TicketConditionalCreationStatusPlugin from TracHacks.  This plugin allowed me to conditionally set the status of the ticket based on the contents of the “owner” field.  This was actually a pretty slick solution, but for various reasons it didn’t sit well with a lot of people.

The next idea we had was to prevent new tickets from being created with any owner besides the default value of “dev” (our dev team mailing list).  This solution had three main benefits – It worked the same for everyone; no one would ever have a “new” ticket assigned to them; and the “dev” alias was always associated with a new ticket, which has some value in relation to email notifications.

Easy enough, right?  Maybe it’s just me, but I could not find a good way to enforce this.  There’s no native ability to hide fields that aren’t drop-downs, and owner’s not a drop down by default.  It can be made a drop-down, but not in a manner which would allow it to be removed.  There are several owner-oriented plugins, but none of them differentiate between the “new ticket” screen and the “modify ticket” screen.

After a bit of fruitless experimentation I realized that I was overthinking it.  I made the following change:

- <body py:match="body" py:attrs="select('@*')">
+ <body onload='document.getElementById("field-owner").disabled = true;' py:match="body" py:attrs="select('@*')">

et voila!  My default owner value is visible but immutable!

I know this isn’t a rock-solid solution, but in our environment (restricted corporate access) I’m not worried about evil-doers and griefers, I’m just wanting to remind people that they’re not supposed to change the value…