TDD: design later

TDD is freaking awesome!



Seriously though, for most projects, TDD is a great place to start!

Let's go on a TDD journey and see how it affects the design of a project!



So I'm working on a project, I have a reasonable idea of how I'm going to design the project and break up the abstractions. First I choose a central set of entities and begin writing tests for them.

Through doing TDD I guarantee that the project is easily testable, because I have to write tests if I choose the wrong part to test or way to test it my life is going to be painful, so I would quickly reevaluate making the wrong decisions because I'm not having TDD fun I'm having TDD pain. TDD will quickly tell you if you're doing it wrong by being extremely painful to write the tests.

As we go forward I follow my design and TDD enforces the testability of the design, and allows me to quickly refactor the design if I have miscalculated anything! Woohoo!



Another time, I'm building a new project, I have no clue about how to design it. I'm not really sure how it will interact with other systems. What the API should be and really where the abstractions lie internally.

This time I just choose what seems a reasonable place to wrap the logic and begin writing tests, I write the hackiest ugliest code in a single file to satisfy the tests and focus on the quality of the tests, not the actual code. After working on the project for some time it's really going to start creaking, there are tonnes of conditional nested logic and edge cases are starting to become quite painful. But at some point, it becomes quite easy to sit down and come up with a design that actually satisfies the project.

With design in hand, I can sit down bin off the file that I was using for the main code and leverage the tests to confirm that the new design works! GLORY!


Don't agree? Leave a comment!

using errors

Much error handling has to be tailored to the situation that suits it, at a minimum, you will want an overall error handler for an API call to handle unexpected errors and report them to the consumer and probably a log.


sometimes we can do something when an error occurs before it hits the default handler, in this case, we return cached data instead.


One important thing to note is the extra value that is added when using the Error class even when rejecting a promise.


This will mean that when we log out the error in the error handler we also get a stack trace, as when a new error is created it records the current stack.

In some cases, we may not be able to fix the problem but we may be able to add additional information to help us when reading the logs later.


now the log will contain the user id and save us some time when debugging later!

Some error cases may be something that we cannot do anything about but the user can, in this case, we are going to need to tell them about it, this hopefully saves us being contacted by the consumer/user and they can fix their own problem.


In this case, we create an additional error type that extends error so that we can bubble that up to be able to alert the user that their account is disabled.

express error handling

Here are a few ideas for error handling in an express app.

For all cases, you will want every route to handle any errors that can occur so that we can notify the caller and potentially do some logging.

Now, this can become quite tedious and repeated so we can pass the error to expresses default error handler!

This will cause express to pass the error out to the client, which in most cases is probably not what we want, so let's have a look at replacing the default error handling middleware.

This now gives us control to be able to add some default logging and return the status we want as well as and information we want to show to the consumer.

One really important thing to remember is to put the function after you define your route handlers! Or it won't catch the errors. (at the bottom of the file)

There are a few different types of things you might be doing in the routes so here are a few examples of how to pass the errors.

This one just handles synchronous code that throws errors.

In this case, we see an example of how to pass callback based errors.

And then moving into promises, as long as we make sure that we pass the error to next, the error handler middleware will be activated.

More on error handling strategies to come!

async/await in node.js: error handling

So, let's have a quick look at async await error handling!

Like last time we are wrapping our code in a self-executing function so we can use the async keywords.

To handle errors in async functions we use a try-catch block like we normally would in synchronous code. You can see from the example that a rejected promise comes out of our catch block.

As we know from previous posts a promise will be rejected if reject() is called or an error is thrown inside of the promise.

Also if you throw an error in an async function this results in a rejected promise. So you can see now how it all fits together, any promise rejections or errors thrown result in the catch block being used.

Just worthy to note that non-awaited rejected promises will not trigger the catch block but will put up an unhandled promise rejection error.

async/await in node.js

Async await is a syntax that makes your asynchronous code look more like synchronous code, it is basically some nice syntactic sugar built on top of Promises.

You just add the async keyword to your function and or expression and that method will now return a promise! You can then use the await keyword to get the result of the promise!

One thing to note is await can only be used inside of an async function, so if you want to execute the code just in a file you need to wrap it in a self-executing function.

I know what you're thinking, "This is great so I can stop writing those weird promises now right?", well no. Though you can quickly create synchronous-looking code that returns promises in either resolved or rejected state.

This does not allow for the handling of asynchronous code that is not promise based inside of your function.

Unfortunately, this will not work as the function will just return a promise that has undefined or no value. To get around this we need to wrap our callback code in a promise as before.

This can now be used in an async function!

You can also make good use of util.promisify to help you with this!

Just remember it will only work for functions where the parameters match the node pattern of callback last and the callback starts with the error.

You can see in this case that by defining our callback method to the node standard util.promisify happily wraps it in a promise.

In this case, I removed the parameter passed to doTest() this results in a "TypeError: callback is not a function" error being thrown, which will be very confusing if you didn't write the callback code you are wrapping and it doesn't have very good error messages.

More on async/await soon!

Promises in node.js: nesting promise patterns

Join me for a little exploration in some patterns that can be used with promises, we will mainly play around with different ways to pass data through promise chains and how to get what you want out of the other end.

Here we are loading the user information, then passing the id from that object to another two promises. The reason the two on the inside are nested is that they both need access to the user object. This also leads to another potential problem because the inner promises are executed sequentially, and in our case, they don't need to be.

This solves the problem in a pretty neat way, we wrap the two inner calls together in a Promise.all() to get them to execute in parallel. Notice how we take advantage of array destructuring to pass in the results together, this is a very useful little pattern.

There may be a specific requirement for this to be changed into an object rather than an unwrapped array, in this case maybe you could wrap both calls in a method with a .then() map.

It's up to you really where the different conversions and abstractions go, I'd advise being careful of having lots of little functions to wrap things up as it leads to code being hard to follow.

But what if we want to merge all of the data together into a single object, you can, of course, do this by nesting all the promises into one inner promise to share the data, but what if we are trying to avoid doing that?

In this case, we change all of the functions to take and return a context object that is updated by each function, this can seem a desirable way to do things but I would recommend against it. Firstly now every method isn't really a pure loading method its kind of a loadUserMessagesAndUpdateContext() method, while if you use this pattern lots I'm sure people wouldn't be too confused unless they hit issues with the way this is going to execute. If you look at my examples they will update the context object before they resolve. Most of the time this probably won't be a problem, but it could definatly give someone a headache.

Join me next time when we look at async/await!

Promises in node.js: Helper functions

Today we will have a look at some of the cool helper methods provided by the native promise framework. The first couple are just quick helpers to create promises in different states.

These methods quickly wrap values in promises of either resolved or rejected state. This will then execute the path that that promise would normally go down (i.e. .then() for resolved, .catch() for rejected).

In this example we use a made up httpGet function and pass in urls to get an array of returned data. The simplest way to think about Promise.all is that it takes an array of promises and returns a single promse that resolves when all of the promises in the array are complete.

This helper is similar to all except as the name suggests it just returns the first one that completes, this could potentially be a way of implementing timeouts on things, I find myself using Promise.all() much more frequently, but still it is good to understand!

Finally blocks are useful to help you close down resources no matter what happens.

Just a quick note to show that promises can have .then() put on them after they complete and the chain will still be executed. The reason for attaching this inside of another .then() was so that the eventloop would get a chance to fire before attaching the new one.

More promises coming soon!

Promises in node.js: Error Handling

Continuing on with the promise theme, let's have a look at error handling.

When executing this you will get a warning, that starts with:
UnhandledPromiseRejectionWarning: This is a rejected Promise
The warning also tells you that the error was not handled by a .catch() block.

In this case we get a similar message: UnhandledPromiseRejectionWarning: Error: An error happened inside the promise, but not rejected.
Both cases can be handled by adding a catch block to the code.

To add an error handler we just add a .catch() block to the end of the promise execution chain, if reject is called then the parameters passed to reject will come out of the catch handler function.
If an error is thrown then the error will come out of the handler function. So we can see how .catch() blocks are very useful but you need to be careful about nesting promises inside of each other.

In this case, we get an unhandled promise rejection message again, because the .catch() block will not pick up the rejected promise. To solve this we can do a couple of things, in most cases, you can just bubble the promise out of the inner .then() block.

The error handling will now work at a higher level because the promise chains are connected. But sometimes we may want to do some additional handling first.

It is still important to connect the chains by returning the inner promise chain or the outer handler won't fire. When a promise is returned from inside a then block the new promise state will be dependent on the inner one.

In this case, the inner error block will fire but not the outer, as no error was thrown. And the outer .then() block will still fire!

So if you do want to execute both catch blocks make sure to throw an error in the inner one, how much of a good practice this is I'm not sure but it helps to understand what state the promises are in.

Promises in node.js

As mentioned in my callbacks post some of the issues with callbacks can be solved with promises. Also once you understand how the core promise methods work you can use a few tricks to make your code look much nicer.

Firstly we execute our promise returning method "writeFile", this returns a promise that we can call .then() on. The function that we pass into .then() will be executed once the asynchronous code has finished executing much like a callback.

In this example we can see that data can be passed from the promise to the .then() function and then passed into another promise, but doesn't it look like we are going down the same too much indenting route in the code?

One of the cool things about promises is that they pass whatever is returned out of a .then() function out as a promise. Let's take a look at a couple of examples to make this clear.

We can see in this example that the text "TESTING" is returned out and wrapped in a promise, the second .then() function will receive this data as a parameter.

In this example the second block returns another promise already so that promise will be returned from the first .then(), allowing the second .then() to trigger after the first completes. This is a simple yet powerful mechanism that helps us to write more straight line code around promises and mix asynchronous and synchronous then blocks in a way where the intent of the flow is clear. And it should all work nicely as long as we remember to return the promises out of the .then() expressions.

One of the other things you will quickly need to do is create your own promises, you can wrap callback code in promises to make your life easier.

In this example, we have a function that creates a new promise wrapping the fs.writeFile function, in its simplest form a promise is an object that takes an expression with 2 parameters, resolve and reject. We can execute asynchronous code inside of the block and just make sure to call resolve when we are completed or reject in case of error/failure. In this case we implement a callback on fs.writeFile that calls resolve when completed.

We can also use util.promisify(), this function will take a method with a callback and return the same method wrapped in a promise. It should definitely work for all of the core node functionality and probably most other things, but be cautious with non node functions (from libraries), as they could implement a different signature and not work.



So yeah that's pretty awesome! more on promises coming soon! and probably some async/await!

Callbacks in node.js

Dealing with asynchronous execution is a big part of writing code nowadays, there are many different ways languages allow you to do this but I am a big fan of how node.js does it. Your code in node.js only executes in a single thread, this makes reasoning about it so much easier and it uses some simple mechanisms to make a single thread more than enough to run the average web server.

To do this any slow running operation you start will be run on another thread that you have no control over.

This would output "I happen before it is finished" before outputting "finished writing file". This is caused by the callback function being queued for execution on the main event loop as soon as the operation completes. One thing to be careful of is keeping your execution running, this will block anything else from happening in your code.

Because this loop keeps executing it stops the event loop from being able to run anything else and basically takes the web server down until you release control. This is why you should avoid using the sync methods that are provided by many APIs as they will block the event loop while they run.

So this is all well and good, and one of the things I like about this approach is it is reasonably simple to understand and get started using node.js. But as time goes on you will probably find some problems.

You can already see the way that this is going, most people call it callback hell, when the code just keeps indenting as you get more callbacks, and you can manage this by wrapping some of the functionality into your own functions but it can still be hard to manage. Especially when most of your functionality is orchestrating other slow-moving parts.

This doesn't really solve the problem in my opinion as it starts to make the flow of the code very hard to follow jumping around the file and also dealing with lots of callbacks. There is one other problem with the basic callbacks that you will most likely hit before long.

While this is better than some of the ways I have seen to achieve this, it's not particularly friendly to follow and it also executes once at a time, which is great if that's a requirement but not so good if it isn't.

I am by no way saying you shouldn't use callbacks, they are very simple and easy to get started within a new node.js application, but quickly you will probably want to start looking into promises (I'll cover these in a blog soon!). Personally, I usually start off with callbacks and migrate to promises when needed, slight inconsistencies don't worry me too much in this case, but as time goes on I do find the promise interfaces nicer for most things and am mostly using them.

Dictionaries for constructing condition-less logic

While I don't recommend removing all conditional logic from your systems, lots of conditions can often be a sign in my mind that the design does not match the problem being solved. Normally I would expect freshly refactored code to be lower on conditional logic and for it to increase heading up to the next run of the design being refactored.


We can start to move the conditional code out into a dictionary, this makes it more open closed principle compliant and less risky to add new functionality to the system. It's great if you've identified that this area keeps changing.


We can take this further and create registration methods so that if we are in a shared piece of code the upper levels or consumers of the code can add additional functionality that will have access to things defined at a much higher level.


In the long run, some of these patterns can be used with care to reduce the amount of confusing logic that is all in one place. so that the core flow of the application is easier to understand.

Blogging Update

So you may have noticed that my blogging has increased significantly of late, last month I made more blog posts than I had in the previous 4 years. It has been really rewarding and has lead to a lot of interesting conversations with people about some of the topics posted.

Why the sudden increase? Well, I have always wanted to blog more but really struggled to get them out. There were many draft posts in different states of completeness sitting unpublished on my blog, so it was time for a different approach, first downsize posts to make them easier to write, then try to develop a daily habit of shipping a post every day. By reducing my size and quality goals I have managed to produce a lot more content and I can work on increasing the quality over time. Is it really quality if no one sees it?

I've managed to quickly resurrect some of the half-written posts and break them into small series as well by just changing my mindset to at least ship something every day. And while I managed most days last month I started on the second and I did have one day where I was ill and didn't write a blog.

It's also strange because I watch quite a bit of fun fun function and he posts every Monday morning and I remember watching one of his videos and it was quite short and I was like man, way to phone it in. But now I think about it people who make this kind of content are just people that are successfully maintaining a habit of regularly producing content, they're not superheroes! Sometimes it's hard to keep up, like today for example when I'm posting in the evening due to being very sleepy this morning!

Well, hopefully I can keep this going for a bit and maybe settle on a slightly different schedule over time, produce and reflect!