Web Working with Webpack

The Problem

You have a computational intense piece of javascript that is blocking browser interactions and animations and making you question why you even try building desktop class web applications. In my case I was trying to bring spreadsheet-like functionality into an application complete with formulas that referenced other formulas that referenced other formulas to the brink of infinity. I ended up needing to calculate tens of thousands of “cells” to get the data I needed. While these calculations can usually complete in under 500 milliseconds (on a fast browser/computer), a 1/2 second where the browser can’t do anything makes for a less than ideal experience. (Think back in the day when you tried to use async: false on your ajax requests because you couldn’t figure this callback stuff out. Was I the only one to try that? 🤔)

Solution #1 - setTimeout

In a server or native application we’d fire up a new thread to punt all of these computations to. The browser, for better or for worse, is all like #lolThreads. I was not familiar enough with web workers at the time to immediately reach for it so I improvised with the setTimeout trick. It went something like:

function calculate(years, callback) {
  processYear(years, {}, callback);
}

function processYear(years, results, callback) {
  if (years.length) {
    doAllTheThingsForYear(years.shift(), results);
    setTimeout(processYear, 0, years, results, callback);
  } else {
    callback(results);
  }
}

This actually worked quite well. It uses setTimeout to give up control after each year and lets the browser do all the things it needs to do. Although it works well it still felt a little hackish and assumes you have a good seam like processing over a number of years to give up control at. Also, for our tests we don’t want to just give away milliseconds like that because they add up over time.

Solution #2 - web worker

So I let the setTimeout hack ride for a week or two before I listened to this Adventures in Angular episode where they talked about web workers in Angular. They spent a good amount of time trying to come up with real world examples of where web workers would help. As I listened it occurred to me that I was sitting on one of the quintessential examples…a spreadsheet…so it made me give them another look.

When I realized that a web worker loaded an entirely separate script resource I almost gave up, but we’re using webpack so it’s just another entry, right? Kind of. My first attempt was exactly that which was adding another entry with all my web workin code. The problem with that is this other script runs in a completely different thread outside of your page so when it tries to load it has no idea about the webpack module loading. In other words, it didn’t work.

I thought to myself…surely someone has done this before with webpack so off to duckduckgo it was. It turns out that there is an example of how to do this in the webpack repo itself. Unfortunately, there’s not a lot of actual words in this example so you have to have both a good understanding of web workers and webpack to interpret it. The result is going to be a little anti-climatic since my solution looks a lot like the example, but here goes:

My first step was to make the calculator code where I had previously had the setTimeouts synchronous:

function calculate(years) {
  const results = {};
  years.forEach((year) => doAllTheThingsForYear(year, results));
  return results;
}

Simple enough, I like deleting code. Step 2 was to change the way I was calling the calculator since it moved from callback based to synchronous:

Before:

function serviceUsedByComponent($q) {
  return {
    calculate: function() {
      return $q(function(resolve) {
        calculatorService.calculate([2016, 2017, 2018], (results) => resolve(results));
      });
    }
  };
}

After:

function serviceUsedByComponent($q) {
  return {
    calculate: function() {
      return $q(function(resolve) {
        resolve(calculatorService.calculate([2016, 2017, 2018]));
      });
    }
  };
}

$q is Angular’s promise library. I had previously wrapped my interactions with the calculator in a promise in order to support the setTimeout solution.

At this point I’m back at square one with my component that needs the data using a synchronous call into the calculator.

[webpack enters stage right]

making the worker work

function serviceUsedByComponent($q) {
  const Worker = require('worker!../workers/calculator.worker');
  const worker = new Worker();

  return {
    calculate: function() {
      return $q(function(resolve) {
        worker.onmessage = (event) => resolve(event.data);
        worker.postMessage([2016, 2017, 2018]);
      });
    }
  };
}

Here we change it up a little to instantiate a new worker from the worker code loaded by webpack’s worker-loader so don’t forget to npm i worker-loader. Instead of calling the calculator directly we now post a message to the worker with our calculator input and resolve the promise with the data we get posted back to us from the worker. That’s it for calling the worker, now on to the worker itself which is a new file.

working on the worker

require('babel-polyfill');

const calculatorService = require('services/calculator.service');

onmessage = function(event) {
  postMessage(calculatorService.calculate(event.data));
};

The first line imports babels polyfills in case you use anything like Object.assign. Remember that this worker is loaded in it’s own thread/context so you don’t get anything from your regular webpack bundle. The rest of the file is receiving the message we posted from the main thread and posting back the results of the calculations.

webpack.config.js

The only thing I changed in my webpack config was taken from the example itself which sets the output filename for the worker:

worker: {
  output: {
    filename: 'calculator.worker.js',
    chunkFilename: '[id].calculator.worker.js'
  }
},

ship it

There you have it, webpack saves the day again.

errors

I left out error handling in my examples for brevity. I’m starting off with a nodejs style err property on my postback messages, but it appears you can get any errors back with an onerror callback - details here.

browsers

Browser support is all I needed it to be, ymmv - caniuse

performance

Performance wasn’t any better in my case and it may be slightly worse. I don’t think a user is going to be able to tell the difference between 500ms and 700ms. The more important part here was getting these computations off of the ui thread. I’m sending a fairly complex object which may be helped with Transferable Objects, but I’m calling it good for now. ¯\_(ツ)_/¯

lazy loading

An unforeseen benefit in our situation was that this provides lazy loading qualities. Our calculator is relatively big so it’s nice to not have that in the initial load.

~ Side Products

Beyond my woes of ditching Progressive Enhancement for an SPA architecture, I think my biggest takeaway is: Users expect rich interfaces.

Dave Rupert talking about building a side product. Going the full stack rails way is exactly what I would have done (and have done) in the past to build an MVP. My feelings have been starting to shift to API first recently. I think this allows you to iterate more quickly on the client(s) and not be tied to an html producing backend server that you have to meticulously sprinkly javascript on to get the experience that users expect. I’d probably even start with the client first to drive out exactly what is actually needed from the backend. This is what my current team did with our product before I started and I think it has worked out well so far.

Permalink

re-rejecting promises

Let’s pretend that you have a bit of code like below…

    
angular.module('demo', [])

  .controller('LoginController', function(loginService) {
    var vm = this;
  
    vm.login = function login() {
      loginService.login(vm.username, vm.password)
        .then(function () {
          setBackground('green');
        })
        .catch(function () {
          setBackground('red');
        });
    }
  
    function setBackground(color) {
      angular.element(document.body).css({'background-color': color});
    }
  })

  .factory('loginService', function ($http, $q) {
    return {
      login: function login(username, password) {
        return $http.post('whatevs', {u: username, p: password})
          .then(function (response) {
            return response.data.token;
          })
          .catch(function (reason) {
            console.log('wtf - ' + reason);
          })
      }
    };
  });
    
  

See the Pen broken promise by Darrin Holst (@darrinholst) on CodePen.

If you’re unfamiliar with angular then the gist of it is that you have the LoginController that gets the login function called when the Login button is clicked. The controller uses a service called loginService (ignore that part about the service being defined with a factory call) that gets injected via the magical powers of angular. It uses that service to call an endpoint and returns a token on success and logs a message on failure. Back in the controller we set the background color to green on success and to red on failure. I know, right? That’s web 2.0 at its finest.

Assuming that the http call to whatevs fails; what do you expect to happen? As of before yesterday my guess would be the message would get logged in the service and then the controller would set the background to red.

It’d be a pretty boring post if that is what actually happened so go ahead and click Login below to see for yourself.

See the Pen broken promise by Darrin Holst (@darrinholst) on CodePen.

Green, amirite?

We can see from the console that we got the error logged.

console

The problem ends up being in our catch in the service. If you think of it like a traditional try/catch in other languages it makes sense. We caught the “exception”, but we didn’t “rethrow” it. The fix is to return a rejected promise in our error handling so the rejection gets propagated up the promise chain.

Notice the addition of the return $q.reject(reason); line in the service ($q is angular’s promise implementation).

    
angular.module('demo', [])

  .controller('LoginController', function(loginService) {
    var vm = this;

    vm.login = function login() {
      loginService.login(vm.username, vm.password)
        .then(function () {
          setBackground('green');
        })
        .catch(function () {
          setBackground('red');
        });
    }

    function setBackground(color) {
      angular.element(document.body).css({'background-color': color});
    }
  })

  .factory('loginService', function ($http, $q) {
    return {
      login: function login(username, password) {
        return $http.post('whatevs', {u: username, p: password})
          .then(function (response) {
            return response.data.token;
          })
          .catch(function (reason) {
            console.log('wtf - ' + reason);
            return $q.reject(reason);
          })
      }
    };
  });
    
  

See the Pen rejection by Darrin Holst (@darrinholst) on CodePen.

Try ‘er again…

See the Pen rejection by Darrin Holst (@darrinholst) on CodePen.

Like most everything else, it makes sense once you know how it works and is completely baffling when you don’t. #themoreyouknow

Check out this excellent post to get a better understanding of promises if you don’t already know all the things. In particular, I found this very helpful…

Every promise gives you a then() method (or catch(), which is just sugar for then(null, …)). Here we are inside of a then() function:

somePromise().then(function () {
    // I’m inside a then() function!
});

What can we do here? There are three things:

  1. return another promise
  2. return a synchronous value (or undefined)
  3. throw a synchronous error

That’s it. Once you understand this trick, you understand promises.

XL stacktraces in your angular tests?

I’ve been stumbling through learning angular the last several weeks and I ran into a frustrating problem today where I was getting an excessivly large stacktrace printed in my test output. A contrived example of my test looks like…

it('should not have a giant stack trace', inject(function ($document) {
  expect($document.find('.fumullins')).to.have.length(1);
}));

Nothing fancy there. It’s just looking to see if an element with a certain class exists on the page. I would expect something like the following to be in the last few lines of my test watch window so I know the test failed for the right reason.

expected { Object ... } to have a length of 1 but got 0

I was presented with the following very helpful output instead…

the object {
  "line": 875
  "message": "expected { Object (length, prevObject, ...) } to have a length of 1 but got 0"
  "name": "AssertionError"
  "sourceId": 207567360
  "stack": "AssertionError: expected { Object (length, prevObject, ...) } to have a length of 1 but got 0
    at base/node_modules/chai/chai.js?d8de1f708fb0a86a7411a0eba65bb24c852addaa:875
    at assertLength (base/node_modules/chai/chai.js?d8de1f708fb0a86a7411a0eba65bb24c852addaa:1939)
    at base/node_modules/dirty-chai/lib/dirty-chai.js?4ec10c65927a645a87817aa9142cd7df2b9263cb:127
    at base/node_modules/chai/chai.js?d8de1f708fb0a86a7411a0eba65bb24c852addaa:5289
    at assert (base/node_modules/chai/chai.js?d8de1f708fb0a86a7411a0eba65bb24c852addaa:4052)
    at base/spec/app/common/secretFormatter/secretFormatter.directive.spec.js?9dd0fd4d1504a3a1a0b79d0c9533131e13223438:15
    at invoke (base/src/bower_components/angular/angular.js?1061b10c86c20aae22472741139f553995ce2951:4203)
    at workFn (base/src/bower_components/angular-mocks/angular-mocks.js?bd422b0634824fd1aec4b70bdad3b63fc7e1f519:2436)
undefined"
  "stackArray": [
    {
      "line": 875
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/chai/chai.js"
    }
    {
      "function": "assertLength"
      "line": 1939
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/chai/chai.js"
    }
    {
      "line": 127
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/dirty-chai/lib/dirty-chai.js"
    }
    {
      "line": 5289
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/chai/chai.js"
    }
    {
      "function": "assert"
      "line": 4052
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/chai/chai.js"
    }
    {
      "line": 15
      "sourceURL": "/Users/dholst/Projects/secret/spec/app/common/secretFormatter/secretFormatter.directive.spec.js"
    }
    {
      "function": "invoke"
      "line": 4203
      "sourceURL": "/Users/dholst/Projects/secret/src/bower_components/angular/angular.js"
    }
    {
      "function": "workFn"
      "line": 2436
      "sourceURL": "/Users/dholst/Projects/secret/src/bower_components/angular-mocks/angular-mocks.js"
    }
    {
      "function": "callFn"
      "line": 4562
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "line": 4555
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "line": 4974
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "line": 5079
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "function": "next"
      "line": 4899
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "line": 4909
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "function": "next"
      "line": 4844
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "line": 4871
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "function": "done"
      "line": 4518
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "function": "callFn"
      "line": 4573
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "line": 4555
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "function": "next"
      "line": 4872
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "line": 4871
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "function": "done"
      "line": 4518
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "function": "callFn"
      "line": 4573
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "line": 4555
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "function": "next"
      "line": 4872
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "line": 4876
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
    {
      "function": "timeslice"
      "line": 6483
      "sourceURL": "/Users/dholst/Projects/secret/node_modules/mocha/mocha.js"
    }
  ]
} was thrown, throw an Error :)

Wtf? No, the effing smiley face at the end doesn’t make me feel better, but thanks mocha. That was 138 lines to scroll up to find out that my test failed for the reason I thought it did in case you didn’t count. The good news with modern day javascript development is that there are very few tools in the chain to dig through to figure how to get rid of this giant stackArray thing. Was it karma or was it mocha or was it karma-mocha or maybe chai or possible karma-chai or maybe phantomjs has something to do with it. At this point I was all like (╯°□°)╯︵ ┻━┻.

I was pretty certain that it was a mocha issue and I did find a nice stack trace cleaner in the process of trying to figure it out. That helped with the stack, but it didn’t help with that weird stackArray thing.

Some more googling and some greping finally landed me on this in angular-mocks.js…

try {
  injector.invoke(blockFns[i] || angular.noop, this);
} catch (e) {
  if (e.stack && errorForStack) {
    throw new ErrorAddingDeclarationLocationStack(e, errorForStack);
  }
  throw e;
} finally {
  errorForStack = null;
}

var ErrorAddingDeclarationLocationStack = function(e, errorForStack) {
  this.message = e.message;
  this.name = e.name;
  if (e.line) this.line = e.line;
  if (e.sourceId) this.sourceId = e.sourceId;
  if (e.stack && errorForStack)
    this.stack = e.stack + '\n' + errorForStack.stack;
  if (e.stackArray) this.stackArray = e.stackArray;
};

This is where the light bulb went off. If you go back up to the test you notice that I wrapped my assertion in an inject call. I did this just out of convenience to get a dependency injected closer to where it was needed in the test.

tl;dr

Don’t wrap your assertions in an inject function.

I put my dependencies in a before block like so…

var $document;

beforeEach(inject(function (_$document_) {
  $document = _$document_;
}));

it('should not have a giant stack trace', function () {
  expect($document.find('.fumullins')).to.have.length(1);
});

and #boom, a more sane looking message

PhantomJS 1.9.8 (Mac OS X) secret-formatter should not have a giant stack trace FAILED
        AssertionError: expected { Object (length, prevObject, ...) } to have a length of 1 but got 0
            at /Users/dholst/Projects/secret/node_modules/chai/chai.js:875
            at assertLength (/Users/dholst/Projects/secret/node_modules/chai/chai.js:1939)
            at /Users/dholst/Projects/secret/node_modules/dirty-chai/lib/dirty-chai.js:127
            at /Users/dholst/Projects/secret/node_modules/chai/chai.js:5289
            at assert (/Users/dholst/Projects/secret/node_modules/chai/chai.js:4052)
            at /Users/dholst/Projects/secret/spec/app/common/secretFormatter/secretFormatter.directive.spec.js:21

Hope that helps somebody.

What Remote Means To Me

I have been fortunate enough to work remotely full time for going on 5 months now and I have a few thoughts on it. Most of my thoughts mirror what Jason Fried and DHH said in Remote. It’s a really short read that gets to the point much like their other books. I recommend it for not only the person that wants to work remotely, but also for those people that manage remote workers. Management of remote workers plays a big part in the success or failure of it.

Another book I would recommend is Quiet: The Power of Introverts in a World That Can’t Stop Talking. Although I don’t have any statistics, I think that introverts lean more towards enjoying and thriving in a remote environment. This book is good (and probably better) for the extrovert that doesn’t understand us hermits. I learned a lot about what drives extroverts from reading it.

Remote work is not for every type of person or every type of business, but given the right combination of those two and it can be a satisfying and productive work environment.

The Commute

Remote work means that an hour is not spent in my metal coffin (bonus points if you identify the movie reference Johnny) every day. My math shows 1 hour x 5 days x ~46 weeks x ~40 years = over a calendar year in your car. I would rather use that time getting my 10,000 steps and saving the tremendous amount of money that is sunk into transportation costs. The reduced stress from the lack of a white knuckle drive in a typical Iowa winter is a bonus too. 90% of drivers believe that they are a better than average driver.

Hold my beer and watch this shit. WHEEEEEE!

Family and Time Management

Working remotely means being able to help get my kids ready for school. It means being able to take them to school when my wife is sick. It means helping my 5 yr old with a puzzle for a few minutes at 10am.

I don't condone the eating of smurfs. They are calorie dense and nutrient poor.

The time I actually do work stays the same, I just get to choose how to slice up my day now. I’m not stuck with 8-12, 1-5 with 2 15 minute smoke breaks.

Health and Nutrition

Remote means giving a shit about what foods I put into my body and not settling for the usual lunchtime crap. Yes, I’m aware that I can pack a lunch, I’ve tried it…a lot. I’m also aware that it’s a heck of a lot easier to just go upstairs and make something at lunchtime instead of guessing what I’ll want 6 - 18 hours ahead of time. Save money, eat healthier, stay away from people making food for you for the vast majority of your meals.

Lunch is an event. Ask me about intermittent fasting sometime.

Pajama Time

For all the things that remote work means to me, it does not mean working in my jammies while watching The Price is Right. I do actually get up at the same time every morning, shower and put on big boy clothes usually beginning around 5:30am. That said, all bets are off at 4pm when Dr. Oz comes on.

Dr. Oz is going to look sweet on this baby!

There is a stigma that is attached to remote workers that we work less. The thing is we know this so that tends to make us work more than we normally would at the office. 40 hours is an important number that shouldn’t be exceeded. Watch out for overwork, it’s as bad or worse than underworking. RescueTime is a good way to keep track of where you spend your time. Give people the tools they need, stimulating things to work on and sit back and watch the human nature to want to do great things work.

Carving Out Your Space

It also does not mean lounging on the couch with my laptop. I have a dedicated space where I stand roughly 90% of my working day.

Rustic theme with custom hardwood floors...er desk

This is important and it’s one of the reasons remote work failed for me when I tried it over a year ago. I would work from coffee shops or at the kitchen counter or on the couch. Once I setup the space where I work it’s like a switch is flipped in my brain that says ‘I do work stuff here’. I rarely do any personal stuff in my work space which keeps the work/home boundaries more defined.

background-color: transparent;

Transparency is key to a successful remote working environment and more important if you are the minority in your company as a remote worker. Everyone should know what everyone else is working on. I am still working on this piece, but things like hipchat and face to face communication on a regular basis should be incorporated. (Yes, I do get out of the house. How else would I get crickets for the bearded dragon and my Trader Joe’s coconut oil?)

Working remotely has a long way to go to become mainstream (if it ever does) especially in the era that we’re in now where the top of the ladder still believes in the factory worker model. It is changing though and it’s a way of working that I’ve found to be the most sustainable and satisfying for me. All I am saying is give remote a chance.

P.S.

There is a whole other topic that I intentionally skirted. How do Agile practices fit into all of this? I ignored it for now because I haven’t come to a conclusion yet. It may need a rethinking.