Code  
o'clock

Canvas-based blur detection with JavaScript

Inspector Bokeh is an experimental JavaScript library to measure blur in images, created during the DaWandathon 2014. This post illustrates the chosen algorithm, based on a paper by Marziliano et al., and expands on some image processing basics.

Motivation

Shop owners on DaWanda's marketplace upload around 40,000 product images of varying quality every day, ranging from snapshots taken with camera phones to professionally edited, studio-quality pictures.

Schöne Bankerslampe im romantischen Jugendstil banker's lamp by Palazzo Int

Apart from requiring certain file formats and dimensions, we currently have no means of assuring the quality of these images. Some of our listed products are manually curated in certain sections of the website, but it wouldn't be possible for us to classify every uploaded image this way.

The goal of this project was to experiment with automated quality checks in the browser. This could provide feedback to users and potentially prompt them to upload images of higher quality.

'Why is this image bad?'

Initial research was conducted to answer this question by clustering images based on different types of degradations (I basically sat around for a day and bothered our backend with countless queries, looking for the worst images I could find).

Some quality problems are fairly easy to detect and solve (e.g. over- or underexposure, low contrast), while others would require complex analysis (e.g. cut off edges of objects, hard-to-read text on images). By far the most frequent cause was blur, so I focused on measuring this particular effect.

Measuring blur

There are several distinct types of blur, but for the sake of simplification I was looking for a general algorithm which only processes one given image. The orginal, unblurred version is assumed to not be available (no-reference blur detection). Researching the state of science on this topic, I compiled this chronologically sorted overview of papers:

  1. A no-reference perceptual blur metric (2002)
  2. Blur Detection for Digital Images Using Wavelet Transform (2004)
  3. Blurred Image Detection and Classification (2008)
  4. Image partial blur detection and classification (2008)
  5. An improved no-reference sharpness metric based on the probability of blur detection (2010)
  6. No-reference image blur assessment using multiscale gradient (2011)

Given the limited amount of time and working knowledge of statistical methods, I chose to implement the algorithm presented in the first paper from 2002 Marziliano et al.. It is relatively simple and based on measuring the width of vertical edges. Given a high enough contrast, it can be observed that edges get smoothened out and appear wider than in the original, if blur occurs.

Detecting vertical edges

As a preparation, the image is converted to greyscale. We only care about the overall contrast in light intensity, not about edges of a particular color. The main work is then done by a convolution filter. This algorithm examines the surrounding pixels of each individual pixel and calculates a new value with the weighted sum of its neighbors. Different kernels/matrices produce different results such as sharpening, blurring, and edge detection. The following 3×3 kernel is also called sobel operator and detects only vertical edges:

-1  0  1
-2  0  2
-1  0  1

For each pixel C, we look at the surrounding pixels px1 to px8:

px1 px2 px3
px4  C  px5
px6 px7 px8

The intensity values for all these pixels are multiplied with their respective weights and added up. This results in a new value for the center pixel C. To demonstrate this, assume the following actual values of a center pixel (87) and its eight neighbours in a greyscale image:

 13  120  111
 46   87  104
  9  114    5

Now we plug the numbers into the sobel operator:

(px1 * -1) + (px2 * 0) + (px3 * 1) +
(px4 * -2) + (C   * 0) + (px5 * 2) +
(px6 * -1) + (px7 * 0) + (px8 * 1)

After removing the zeros and filling in the actual pixel values, we get:

-13 + 111 - 92 + 208 - 9 + 5 = 210

The sobel operator turned the original center pixel with value 87 into 210.

The end result of edge detection, when it is done in both horizontal and vertical direction, combined with smoothing, is an image like the one below:

Measuring the width of vertical edges

The edges we detected are evident to a human seeing the greyscale image, but we still need to make them ‘machine visible’ - i.e. find out exactly where edges begin and end. Marziliano et al. used local minima and maxima to determine this, but didn't fully reveal their process to get them, at least to my best knowledge and understanding. So I chose simpler criteria:

  1. an edge begins when the intensity value is anything greater than zero (a black pixel, no edge found in the original image).
  2. Given that we're processing the pixels in a row from left to right, the edge continues as long as the intensity values keep increasing.
  3. The edge ends as soon as the subsequent value decreases.

To not count edges caused by noise in the original image, we discard intensities below a certain threshold. Usually the clearly visible edges in an image have enough contrast around them to end up as 100% white pixels after edge detection.

Core algorithm in JavaScript

// For each row of pixels
for (y = 0; y < height; y += 1) {
    // Reset edge marker
    edgeStart = -1;

    // For each pixel in the current row of pixels
    for (x = 0; x < width; x += 1) {
        // Intensity value of current center pixel
        value = pixels[y][x];

        // Edge is still open
        if (edgeStart >= 0 && x > edgeStart) {
            // Intensity value of previous pixel
            oldValue = pixels[y][x - 1];

            // Intensity is not increasing anymore, the edge ended
            if (value < oldValue) {
                // Only count edges above a certain intensity
                if (oldValue >= edgeIntensThresh) {
                    // Calculate edge width
                    edgeWidth = x - edgeStart - 1;
                    // Count total number of detected edges
                    numEdges += 1;
                    // Add to sum of all edge widths
                    sumEdgeWidths += edgeWidth;
                }
                // Reset edge marker
                edgeStart = -1;
            }
        }
        // Beginning of a new edge detected
        if (value == 0) {
            // Remember where the edge started
            edgeStart = x;
        }
    }
}

The final blur score is the average width of a vertical edge, in relation to the width of the image:

avgEdgeWidth = sumEdgeWidths / numEdges;
blurScore = avgEdgeWidth / width * 100;

A high blur score suggest a high amount of blur in the given image.

Inspector Bokeh at work

Running measure_blur.js as a Node.js module on an image and its blurred versions gives the following output:

  1. Original image (378 × 528 pixels)
  2. Gaussian blur with radius of 2 pixels applied
  3. Gaussian blur with radius of 8 pixels applied

Conclusion

The algorithm still needs a lot of fine tuning and is far from anything production-ready. The major problem is that noise/grain is wrongly detected as edges. To ease this, the current implementation applies a low-radius gaussian blur on input images of lower resolution. Despite these limitations, Inspector Bokeh can be used to automatically detect extreme cases of blurred images when a threshold for the resulting blur score is carefully chosen.

Responsive ratio keeping box

How to create CSS only responsive divs that keep their ratio.

responsive ratio video

Discussion

Sometimes you would like to include a third party widget like a video into your page. Usually iframes are used for including such things, but iframes are a technique back from the old days and have a fixed width and height - thus not allowing them to react in a responsive way.

Most YouTube videos have a ratio of 16:9. If you change your device width you have to calculate a new height for the box. You can do that with JavaScript, but there is also a flexible CSS-only solution, which we will show you here.

Groundwork

First we need to know some mathematical basics: how you can calculate the percentage ratio height to width of a box. If the percentage height of the box is relative to its width, the box will keep its ratio on all devices.

The mathematical formula to calculate the percentage from ratio a:b is:

(b ÷ a) * 100

For example, given the ratio 16:9, the calculation is

(9 ÷ 16)* 100 = 56.25%

How does calculating this help us? Here comes the magic of CSS. If you have a div or any other box element with a given width and you use percentage value for the padding of this box, the padding will be relative to the width of the box. Therefore, if your box is 500px wide, 56.25% would be 281px ( or 282px - depends on the browser) high.

This still works when you have a percentage width value on your box, because this width will be relative to the surroundings. If, for example, your box has a width of 100%, the padding will be relative to whatever 100% width of the box means.

Tying it together

How does this knowledge fix our initial problem? When you add a large amount of padding-top to a box, the content will be simply pushed down. Therefore, we need to wrap things up a little bit..

Here's the markup you'll need for your responsive box:

<div class="ratio-box ratio-16-9">
  <iframe class="ratio-content" width="560" height="315" src="https://www.youtube.com/embed/5lH8X39iF8w" frameborder="0" allowfullscreen>
  </iframe>
</div>

Now you have a ratio-box wrapper around the ratio-box content

Now let's add some CSS to it:

.ratio-box {
  position: relative;
  width: 100%;
}

.ratio-16-9:before {
  content: " ";
  display:block;
  padding-top: 56.25%
}

.ratio-box .ratio-content {
  position: absolute;
  top: 0;
  left: 0;
  width: 100%;
  height: 100%;
}

Getting more advanced

This is already working very well, but how about writing a SASS mixin which allows you to call this every time you need a new ratio?

@mixin ratio-box($a, $b) {
    &:before {
      content: " ";
      display:block;
      padding-top: percentage($b / $a);
    }
  }

With this mixin you can now create all kind of ratio-classes!

RubyConfLT

This year we proudly sponsored RubyConfLT. It took place in Vilnius and a bunch of us went there. We got to meet plenty of great local and visiting people, even some friends from Berlin. It was a one day conference, not super big, but great atmosphere and we're very happy that we went there.

Some moments from there:

Audience Nick Sutterer Ben Lovell Tobias Pfeiffer Bozhidar Batsov Arne Brasseur DaWanda Luca & Tomas Badge

Fun with tables

One possible solution for responsive tables

Challenge

When it comes to tables there is a history of love and hate. For a long time, tables where incorrectly used to solve layout problems. This was fixed by CSS 2, which introduced several possibilities for creating layouts. After IE caught up on CSS table-display options, tables could finally be used as intended: displaying tabular data in an organized and readable way.

Unfortunately this time of happiness for tables did not last. Now with CSS 3 and responsive design are prevalent, tables are again not flexible enough to serve tiny mobile displays. You can squeeze them down to be very small, but this makes them unreadable in most cases. Additionally, horizontal scrollbars are not the best solution as you loose the ability to quick glance at table content.

Solution

Here at DaWanda we've had a responsive design since summer 2013. This is our CSS-only solution for displaying tabular data on mobile.

UX perspective

Mobile first

mobile view of the fake table

From UX perspective we start with the mobile view. The mobile table focuses on giving an overview of each row in turn. The most important information is in the first row and should be self-explaining without a requiring heading. The following rows are have a heading and a value. The last row is for displaying possible actions on the table. They should also be self-explaining without heading.

Desktop view

desktop view of the fake table On the desktop we rotate table 90°. Each mobile table-box becomes one row inside our table. We also have now the typical header row with the title of each column. These are desktop tables everyone is familiar with.

Technical side

Markup on mobile

It's clear to see this is nothing you can achieve with a native table element, because you can not turn a td into a tr. Instead we're using divs to build our table.

<div class="fake-table-autowidth">
    <div class="ft-row">
      <div class="ft-cell cell-title">
        EBook - my simply + endless summer - XS - XL
      </div>
      <div class="ft-cell">
        <span class="cell-title visible-xs-inline">Filetype</span>
        PDF
      </div>
      <div class="ft-cell">
        <span class="cell-title visible-xs-inline">Filesize</span>
        9 MB
      </div>
      <div class="ft-cell">
        <a href="http://de.dawanda.com/shop/schaumzucker" target="_blank">schaumzucker</a>
      </div>
      <div class="ft-cell">
        <span class="cell-title visible-xs-inline">purachsed at</span>
        10.05.2014
      </div>
      <div class="ft-cell">
        <button class="btn primary">download</button>
      </div>
    </div>
</div>

We just nest as we would do with a vanilla table. We have one row which contains cells. The titles we show only in mobile have a class which displays them as inline elements.

Additional markup for desktop

For desktop view we have to add the table-headline with the titles. It's just another row with cells

<div class="ft-row ft-h-row">
    <div class="ft-cell ft-h-cell">
      Productname
    </div>
    <div class="ft-cell ft-h-cell">
      Type
    </div>
    <div class="ft-cell ft-h-cell">
      Filesize
    </div>
    <div class="ft-cell ft-h-cell">
      Shop
    </div>
    <div class="ft-cell ft-h-cell">
      Purchased at
    </div>
    <div class="ft-cell ft-h-cell">
      <span class="text-hide">Download</span>
    </div>
  </div>

CSS magic

Now let's look at the magic part. We start again mobile first

.ft-cell {
    padding: 5px;
  }
  .ft-cell .cell-title {
    font-weight: bold;
  }
  .text-hide {
    font: ~"0/0" a;
    color: transparent;
    text-shadow: none;
    background-color: transparent;
    border: 0;
  }
  @media (max-width: 768px) {
    .ft-h-row {
      display:none;
    }
    .ft-cell:nth-child(even) {
      background-color: #fff;
    }
    .ft-cell:nth-child(odd) {
      background-color: #F8F8F8;
    }
    .ft-row {
      margin-bottom: 1em;
    }
    .xs-inline-50 {
      display: inline-block;
      width: 49%;
    }
  }

We set general styles outside of a media query like font-weight of title elements or padding for the cells. Specific element styles go into our mobile breakpoint media-query which fits to a maximum of 768px. First we hide the table-header row, then we add zebra-colours to each cell. The cells which include a cell-title are set to display:inline-block and width:49% so the title and content are adjacent.

Here is the CSS part of the desktop media-query.

@media (min-width: 768px) {
    .visible-xs {
      display: none;
    }
    .fake-table {
      display: table;
      width: 100%;
    }
    .ft-row {
      display: table-row;
    }
    .ft-row:nth-child(even) {
      background-color: #fff;
    }
    .ft-row:nth-child(odd) {
      background-color: #F8F8F8;
    }
    .ft-row.ft-h-row {
      background-color: grey;
      border-bottom: 1px solid grey;
      margin-bottom:0;
    }
    .ft-cell {
      display: table-cell;
      vertical-align: middle;
      padding-left: 5px;
      padding-right: 5px;
    }
    .ft-cell.ft-h-cell {
      font-weight: bold;
    }
   }

We hide the inside title selected by the class visible-xs and build up our CSS table. we add also zebra-colours to the row and set the font-weight of our heading cells to bold.

Conclusion

Here you can find a demo of the table. This solution is lightweight and straightforward. We do not have to duplicate the content, only additional header titles if they're needed to understand the content. Our solution of course has some negative aspects. It loses the semantics associated with the table elemen and it still needs some duplication. Additionally, tables with many rows occupy a lot of space on mobile. This could be solved by toggling the table contents so that you only see the first column of each, row and on expand you would show the hidden content.

Spring Cleaning

Think of your desk at home or in your office. The desk you would put your computer on and work with. Is it empty or are there lots of things that made it there at some point and were never removed? Do you feel like cleaning it up or better postpone it?

The problem

Now think of your software project. Did it happen, that parts of your codebase got off your mind? Or worse, you knew about weak and obsolete parts but you never found time to reorganize and improve them? Contrary to non-developer opinion, after you ship code it is not gone for ever. The opposite is the case: the code keeps influencing all the work you do afterwards. Consequently this can lead to team frustration and stress in the long run. For example, your codebase can become hard to understand and maintain if it’s untested or non-DRY.

Cartoon of people cleaning up code

The start

At the end of 2014, we developers at DaWanda had our yearly team health retrospective. The outstanding issues with our codebase were the main factors for slow development cycles, as well as the general happiness of the development team. We requested to have a dedicated sprint for cleaning up the major things.

The first sprint of the year looked perfect for this kind of project as we had less stress with our daily routine. Our request was granted with the condition of having a list of tasks we wanted to tackle. Everybody could add topics that caused pain which were then prioritized by the core dev team.

The tasks

We knew from daily experience, and especially all through Christmas, where the biggest pain points and improvement opportunities were. Since 2013, we shifted from a fairly organically-grown codebase to a new, easy maintainable infrastructure and stack. But the legacy codebase keeps influencing us.

Here is a quick overview of what we did in the Clean-Up-Sprint:

  • Big parts of the legacy app were migrated and refactored to the new stack. The objective was not only to copy over the code, but to reimplement, refactor and fully test it. Through this, we could avoid making the same mistakes and rethink our old software design.

  • Some huge parts of the legacy app were removed either because they were no longer needed or they were re-implemented in the new stack. This was important because we wanted to have less old components to maintain.

  • We also took this opportunity to refactor our new stack. Files that grew considerably in the previous months were split, following the principle of separation of concerns. All of these changes were made in small iterations and in a test driven way.

  • A new gem was created to meet our requirements on HTTP resources.

  • We discussed how to improve our Java-based caching solution and ended up changing its garbage collection strategy with the help of JVM monitoring tools to measure results. Then we improved the way our caching solution was purging stale data from Memcached. As a result, our users can interact with the DaWanda webpage significantly faster than before.

  • Moreover, we discussed new external services, introduced them or upgraded old versions.

  • We cleaned up our styleguide and removed duplications in order to prevent confusion in the future. We approached our goal to have all our components with their styles in this one place, as well as giving this app our corporate style.

  • Finally, we tidied our Github repositories and removed obsolete ones. Furthermore, a big bunch of stale branches were thrown out. Open pull requests were reviewed and also reduced to only the most recent ones.

The methods

From the beginning, pair programming was intended and it was clear that it would become important. After some initial challenges, the team were focused on it. On average, every team member paired with 2 to 3 other team members in this sprint. Some pairs worked together for days, which can be exhausting, but also very productive.

Many people do not realize that most of the time we spend solving software problems is not only about finding a solution, but also about reading and understanding the code and the problem itself. The part of actually writing the code is only a fraction of that time. Therefore with two minds thinking of a solution, the problem can be solved faster, with less mistakes and higher quality.

Even team members working remotely paired successfully. One pair completely refactored a controller of our frontend app. Following the refactoring, they raised our CodeClimate rating from F to A, and there was full test coverage - an amazing result!

Cartoon of 2 people giving thumbsup on code climate result

When team members were asked what they liked the best on the Clean-Up-Sprint, the majority mentioned the pair programming. Moreover, they mentioned the sharing of knowledge. It was a prime moment to ask other colleagues questions and they took the time to explain. We learnt a lot of new facts about our applications and gained new skills. We also rethought the software structure and certain implementations from the past to get an inspiration on what to do differently in the future.

We organized our sprint via JIRA (what we use also for our normal sprints), to keep roughly track of estimated effort, tasks completed, and the actual effort taken. Looking back, the feasibility of tasks should have been defined clearly to avoid tasks that cannot be fully completed within these two weeks, or tasks that leave room for improvement.

The outcome

After the two weeks of our Clean-Up-Sprint, we had done a lot to improve our codebase. Many overdue tasks were tackled. We facilitated work processes for our team and reduced many pain points for us and our users. Consequently, the happiness in our team increased and we were proud of our shared success. An almost hackathon feeling was created as a result of less meetings and more pair programming.

After the sprint, all team members said that they would recommend to do a Clean-Up-Sprint on a regular basis, and 64% said they reached their goals. The remaining percentage stated that they had reached them partly.

Next time we want to focus on optimizing the organizational process and plan better. The Clean-Up-Sprint also showed us that we would like to integrate this kind of sprint into our working routine. Moreover, we got inspired to do more pair programming and to share our knowledge. Additionally, cleaning out old and nasty parts of the app has become more of a daily habit for everyone.

We can recommend to dedicatedly clean up from time to time!