OpenJS In Action

We recently met with Ian Sutherland, engineering lead and Head of Developer Experience at Canadian fintech startup, Neo Financial. Ian has been with Neo Financial from the very beginning and has seen the engineering team grow from 1 employee to over 150 individuals in the last three years. Ian is also a Collaborator on the Node.js project hosted at the OpenJS Foundation. Watch the full interview:

https://youtu.be/pCfM4_jxH0E

What is Neo Financial?

Neo is a financial technology company that is reimagining how Canadians bank. Their first product was a rewards credit card, they later introduced a top rated high-interest savings account and recently launched Neo Invest, the first fully digital, actively managed investment experience. With Neo Invest your portfolio is actively managed by experts and engages a greater range of asset classes and investment strategies than most competitor portfolios.

Developed with JavaScript First

The Neo Financial web banking portal provides seamless mobile-first interactions for users. The backend of the portal is built entirely using JavaScript and Node.js and powers all of the app’s microservices and transaction processing. Ian and the engineering team decided early on that they would use JavaScript for everything they possibly could in developing their product. Ian shared his opinion that “Node.js is the technology of choice for running JavaScript on the server, so from day one, a decision was made that the team would use Node.js.”

Other factors influenced their decision to work with JavaScript and related technologies like Node.js. JavaScript is currently the most widely used programming language, which they felt would make it easier to scale their team of developers quickly. The language also, put simply, works well for them. Their team finds it easy to containerize apps using Node.js. It’s also easy to build serverless functions written in JavaScript running on Node.js, with no compilation step required. It’s fair to say that Node.js provides excellent performance and scalability, keeping their team’s infrastructure costs low. 

Working with Other Open Source Technologies

Using Node.js and JavaScript for local development has worked well for the Neo Financial dev team. Ian shared that they have a swift development experience where a person can change some code and have the project reload instantly. He also cited the npm ecosystem and the “millions” of packages out there as a benefit, helping his team work very productively. They also use TypeScript and Fastify in all of their services, and webpack indirectly through other frameworks.

Open source packages and plugins help speed up their team’s development work. Chances are, someone has already dealt with a similar problem. These packages make it easier to solve whatever the needs are without recreating the wheel.

A Note on Security

As a financial company, security is top of mind for Ian and his team. The Node.js project has also been focusing more on the security of Node.js itself. As a result, the devs at Neo feel very comfortable running it in production.

Contributing to Open Source

On a personal level, Ian has been involved in open source for several years. He started by making smaller contributions and later got involved in the React community. He eventually became a core maintainer of the Create React App and has been working on that project for the last three or four years. Then, about four years ago, he got involved in Node.js itself, primarily as part of a working group called the Tooling Group. The focus of this group is on making Node.js the best tool it can be for building things like CLI tools, or other tools that might run in a CI or build environment, lambdas, etc. 

As a team, the Neo Financial engineers try to do their part. They’ve open sourced developer tools and GitHub actions and try their best to give back wherever they possibly can. In a big thank you to the open source community, Ian said, “We have an awesome community. People are doing development on open source projects for free as volunteers when they contribute lines of code and fixes and documentation.”

We at the OpenJS Foundation feel the same way. We wouldn’t be anywhere without our contributors and our fantastic community. It was a pleasure speaking with Ian, and we’re grateful for his input as an individual and an engineering team leader. 

Background:

In Wix, as part of our sites’ Server-Side-Rendering architecture, we build and maintain the heavily used Server-Side-Rendering-Execution platform (aka SSRE). 

SSRE is a Node.js based multipurpose code execution platform, that’s used for executing React.js code written by front-end developers all across the company. Most often, these pieces of JS code perform CPU-intensive operations that simulate activities related to site rendering in the browser.

Pain: 

SSRE has reached a total traffic of about 1 million RPM, requiring at times largely more than accepted  production Kubernetes pods to serve it properly. 

This made us face an inherent painful mismatch:
On one side, the nature of Node.js – an environment best-suited for running I/O-bound operations on its single-threaded event loop. On the other, the extremely high traffic of CPU oriented tasks that we had to handle as part rendering sites on the server.

The naive solution we started with clearly proved inefficient, causing some ever growing pains in Wix’s server and infrastructure, such as having to manage tens of thousands of production kubernetes pods.

Solution: 

We had to change something. The way things were, all of our heavy CPU work was done by a single thread in Node.js. The straightforward thing that comes to mind is: offload the work to other compute units (processes/threads) that can run parallely on hardware that includes multiple CPU cores.

Node.js already offers multi-processing capabilities, but for our needs this was an overkill. We needed a lighter solution that would introduce less overhead, both in terms of resources required and in overall maintenance and orchestration.

Only recently, Node.js has introduced what it calls worker-threads. This feature has become Stable in the v14 (LTS) released in Oct 2020.

From the Node.js Worker-Threads documentation:

The worker_threads module enables the use of threads that execute JavaScript in parallel. To access it:

const worker = require(‘worker_threads’);

Workers (threads) are useful for performing CPU-intensive JavaScript operations. They do not help much with I/O-intensive work. The Node.js built-in asynchronous I/O operations are more efficient than Workers can be.

Unlike child_process or cluster, worker_threads can share memory.

So Node.js offers a native support for threading that we could use, but since it’s a pretty new thing around, it still lacks a bit in maturity and is not super smooth to use in your production-grade code of a critical application.

What we were mainly missing was:

1. Task-pool capabilities –
   What Node.js offers?
   One can manually spawn some Worker threads and maintain their lifecycle themselves. E.g.:
   

const { Worker } = require(“worker_threads”); //Create a new worker const worker = new Worker(“./worker.js”, {workerData: {….}}); worker.on(“exit”, exitCode => {   console.log(exitCode); });

We were reluctant to spawn our Workers manually, make sure there’s enough of them at every given time, re-create them when they die, implement different timeouts around their usage and more stuff that’s generally available for a multithreaded application.

2. RPC-like inter-thread communication –
   What Node.js offers?
   Out-of-the-box, threads can communicate between themselves (e.g. the main thread and its spawned workers) using an async messaging technique:
   

// Send a message to the worker aWorker.postMessage({ someData: data }) // Listen for a message from the worker aWorker.once(“message”, response => {   console.log(response); });

Dealing with messaging can really make the code much harder to read and maintain. We were looking for something friendlier, where one thread could asynchronously “call a method” on another thread and just receive back the result.

We went on to explore and test various OS packages around thread management and communication.

Along the way we found some packages that solve both the threadpool problem and the elegant RPC-like task execution on threads. A popular example was piscina. It looked all nice and dandy, but there was one showstopper. 

A critical requirement for us was to have a way for our worker threads to call some APIs exposed back on the main thread. One major use-case for that was reporting business metrics and logs from code running in the worker. Due to the way these things are widely done in Wix, we couldn’t directly do them from each of the workers, and had to go through the main thread.

So we dropped these good looking packages and looked for some different approaches. We realized that we couldn’t just take something off the shelf and plug it in.

Finally, we have reached our final setup with which we were happy.

We mixed and wired the native Workers API with two great OS packages:

• generic-pool (npmjs) – a very solid and popular pooling API. This one helped us get our desired thread-pool feel.
• comlink (npmjs) – a popular package mostly known for RPC-like communication in the browser (with the long-existing JS Web Workers). It recently added support for Node.js Workers. This package made inter-thread communication in our code look much more concise and elegant.
  

The way it now looks is now along the lines of the following:

import * as genericPool from ‘generic-pool’; import * as Comlink from ‘comlink’; import nodeEndpoint from ‘comlink/dist/umd/node-adapter’; export const createThreadPool = ({                                     workerPath,                                     workerOptions,                                     poolOptions,                                 }): Pool<OurWorkerThread> => {     return genericPool.createPool(         {             create: () => {                 const worker = new Worker(workerPath, workerOptions);                 Comlink.expose({                     …diagnosticApis,                 }, nodeEndpoint(worker));                 return {                     worker,                     workerApi: Comlink.wrap<ModuleExecutionWorkerApi>(nodeEndpoint(worker)) };             },             destroy: ({ worker }: OurWorkerThread) => worker.terminate(),         },         poolOptions     ); };

And the usage in the web-server level:

const workerResponse = await workerPool.use(({ workerApi }: OurWorkerThread) =>     workerApi.executeModule({         …executionParamsFrom(incomingRequest)     }) ); // … Do stuff with response

One major takeaway from the development journey was that imposing worker-threads on some existing code is by no means straightforward. Logic objects (i.e. JS functions) cannot be passed back and forth between threads, and so, sometimes considerable refactoring is needed. Clear concrete pure-data-based APIs for communication between the main thread and the workers must be defined and the code should be adjusted accordingly.

Results:

The results were amazing: we cut the number of pods by over 70% and made the entire system more stable and resilient. A direct consequence was cutting much of Wix’s infra costs accordingly.

Some numbers:

• Our initial goal was achieved – total SSRE pod count dropped by ~70%.
  Respectively, RPM per pod improved by 153%.
• Better SLA and a more stable application – 
  • Response time p50 : -11% 
  • Response time p95: -20%
  • Error rate decreased even further: 10x better
• Big cut in total direct SSRE compute cost: -21%

Lessons learned and future planning:

We’ve learned that Node.js is indeed suitable also for CPU-bound high-throughput services. We managed to reduce our infra management overhead, but this modest goal turned out to be overshadowed by much more substantial gains.

The introduction of multithreading into the SSRE platform has opened the way for follow-up research of optimizations and improvements:

• Finding the optimal number of CPU cores per machine, possibly allowing for non-constant size thread-pools.
• Refactor the application to make workers do work that’s as pure CPU as possible.
• Research memory sharing between threads to avoid a lot of large object cloning.
• Apply this solution to other major Node.js-based applications in Wix.