Dave Clements is an open source advocate and is the tech lead and primary author of OpenJS Foundation Node.js training and certification programs. And a big thank you to Nearform and James Snell for allowing the OpenJS Foundation to repost this article.
Foreword
The OpenJS Node Application Developer certification is an evergreen program that stays up to date with advancements in the JavaScript specification, Node.js core, industry trends, and best practices not only to ensure that the examination and training stay relevant but also to help disseminate important information for the Node & JavaScript community.
With that in mind, the following article by James Snell is republished with permission from James and NearForm where the article was first published. We strongly recommend anyone thinking of taking the JSNAD certification read this article and consider the implications. We hope you enjoy it!
The AbortController and AbortSignal APIs are quickly becoming the standard mechanism for canceling asynchronous operations in the Node.js core API.
If you search how to use the Promise.race() API, you’ll come across quite a few variations of the following:
The intent here is straightforward: Start a potentially long-running task but trigger a timeout if that task takes too long to complete. This is generally a good idea, but there are quite a few problems with this common example.
First, although the promise returned by Promise.race() will be fulfilled as soon as the first of the given promises is settled, the other promises are not cancelled and will keep on running. Although the timeout timer did fire, the long-running task is never actually interrupted and stopped.
Second, what happens to the timeout promise if the long-running task completes before the timeout is triggered? The answer is simple: The timer keeps running, and the promise will end up rejecting, still with an unhandled rejection — unnecessarily risking performance issues and possible memory leaks in your application.
To correctly handle this pattern, we need a reliable mechanism for signalling across the two promises, canceling either the timer or the long-running task as appropriate and ensuring that once the timeout is triggered all resources are cleaned up as quickly as possible. Fortunately, Web Platform APIs provide a standard mechanism for this kind of signalling — the AbortController and AbortSignal APIs.
In Node.js, a better way if implementing a Promise.race-based timeout would be:
As with the previous example, two promises are created. However, when each completes, it uses the AbortController and AbortSignal APIs to explicitly signal to the other that it should stop. As long as the code in those is written to support the AbortSignal API, everything just works.
For instance, in the example we make use of the recently added awaitable timers API in Node.js. These are variants of the setTimeout() and setInterval() that return promises.
The awaitable timer API supports the ability to pass in an AbortSignal instance. When the AbortSignal is triggered, the timer is cleared and the promise immediately rejects with an AbortError.
Support for AbortController and AbortSignal is being rolled out across the Node.js core API and can now be found in most of the major subsystems. Before we explore where the API can be used, let’s find out a bit more about the API itself.
All about AbortController and AbortSignal
The AbortController interface is simple. It exposes just two important things — a signal property whose value is an AbortSignal and an abort() method that triggers that AbortSignal.
The AbortSignal itself is really nothing more than an EventTarget with a single type of event that it emits — the ‘abort’ event. One additional boolean aborted property is true if the AbortSignal has already been triggered:
The AbortSignal can only be triggered once.
Notice that when I added the event listener in the example above, I included the { once: true } option. This ensures that the event listener is removed from the AbortSignal as soon as the abort event is triggered, preventing a possible memory leak.
Note that it’s even possible to pass an AbortSignal onto the addEventListener() itself, causing the event listener to be removed if that AbortSignal is triggered.
This starts to get a bit complicated too, but it’s important for preventing memory leaks when coordinating the cancellation of multiple complex tasks. We’ll see an example of how this all comes together next.
Implementing API support for AbortSignal
The AbortController API is used to signal that an operation should be cancelled. The AbortSignal API is used to receive notification of those signals. They always come in pairs.
The idiomatic way of enabling a function (like the someLongRunningTask() function in our examples above) to support this pattern is to pass a reference to the AbortSignal in as part of an options object:
Within this function, you should immediately check to see if the signal has already been triggered and, if it has, immediately abort the operation.
Next, it’s important to set up the handling of the ‘abort’ event before starting to process the task:
Notice here that we are creating an additional AbortController instance whose signal is passed in with the event listener. After we’ve completed the asynchronous task, we trigger that AbortController to let the AbortSignal know that the event handler can be removed. We want to make sure that the listener is cleaned up even if the async task fails, so we wrap the call to taskDone.abort() in a finally block.
It is also important to check if the signal has been triggered between various async tasks the method may be performing. This is important to catch cases where the event may not yet have had an opportunity to be emitted but the operation should still be interrupted.
Using AbortController and AbortSignal
The AbortController and AbortSignal APIs are quickly becoming the standard mechanism for canceling asynchronous operations in the Node.js core API. For example, as of node.js 15.3.0, it is possible to cancel an HTTP request using the API:
Consult the Node.js documentation for more details on exactly which APIs support AbortSignal. More are being added all the time and support may vary across different Node.js major versions.
7 basic tasks Node.js developers need to understand about Kubernetes
This post was written by Michael Dawson, OpenJS Foundation Board Member and Node.js community lead at IBM. This first appeared on IBM Developer.
Kubernetes is fast becoming the leader for deploying and managing production applications, including those written in Node.js. Kubernetes is complex, though, and learning the ins and outs of the technology can be difficult, even for a seasoned developer.
Node.js application developers may not need to manage Kubernetes deployments in our day-to-day jobs or be experts in the technology, but we must consider Kubernetes when developing applications.
As a reminder, Docker and Kubernetes are the foundation of most modern clouds, including IBM Cloud. Cloud native development refers to creating applications that you will deploy in Docker or Kubernetes, and cloud-native applications often conform to the tenants of a 12 Factor application.
In this article, I try to answer the question: “As a Node.js devleoper, what do I need to know about Kubernetes?” I cover a number of key tasks you must complete when building your Node.js application in a cloud-native manner, including:
Building Docker images
Deploying containerized applications
Testing your applications
Health checking
Logging
Gathering metrics
Upgrading and maintaining containerized applications
Building Docker images
Deploying to Docker or Kubernetes requires that you build a Docker image. To do that, you typically start with an existing image and layer in the additional components that you need.
When building Node.js applications, the community provides a number of official Docker images that you can start with. These images add the Node.js binaries to an existing Linux distribution (Debian or Alpine) and offer a way to bundle your application into the image so that it runs when you start the Docker container.
Each time the project publishes a new release, it creates new Docker images. For example, when 12.13.1 was released, the new Docker image “node:12.13.1” was made available.
There are three variants of the Node.js images:
Debian-based images with the core components needed to build and test Node.js applications.
Slim images which are Debian-based images with only the minimal packages needed to run a Node.js application after it is already built.
Alpine-based images for those who need the smallest container size.
There are a number of tags that you can use to specify the image you want. See Docker Hub for the full list.
When choosing your base image, keep in mind:
Alpine binaries are not built as part of the main Node.js releases. This means that they are built from source and there is more risk of regressions.
You should likely use both the standard Debian image along with the Slim image in a multistage build in order to achieve a smaller resulting image size. More on that in the next section
Build your image
Once you’ve chosen your base image, the next step is to actually build the image which bundles in your application. This is done through a Dockerfile. The simplest Dockerfile is as follows:
This Dockerfile simply copies your application code (server.js) into the official image with the node:12.13.1 binaries and indicates that server.js should be run when the image is started. Note that you did not have to install Node.js as it is already installed and in the default path as part of the node:12.13.1 base image.
To build an image from this file, put the file into a directory with your application code and then from that directory run docker build . -t test1:new. This builds a docker image named test1, and with the new tag, based on the contents of the current directory.
While this sounds straightforward, Dockerfiles can quickly become complicated because you want to build the image in a way that only includes the minimum components needed to run your application. This should exclude intermediate artifacts, for example the individual object files created when native addons are compiled. To exclude intermediate artifacts, a multistage build is commonly used where artifacts are generated in one step or image and then copied out to another image in a later stage. Read more about this in the Docker documentation.
Once you have built the image, you need to push it to a registry and then you can run and test your application under Docker or Kubernetes. It’s best to test in as close an environment to the production deployment as possible. Today, that often means Kubernetes.
Deploying your application to Kubernetes
In order to deploy and test your image in Kubernetes you use one or more deployment artifacts. At the lowest level, much of the Kubernetes configuration is through YAML. To deploy and access your application through a port, you need three YAML files.
To start, you need a deployment which specifies which image to use (the one you pushed to the registry earlier). For example:
In this case, it’s specifying through the image: entry that it should use the Docker image named test1 with the new tag (which matches what we built in the earlier step). You would use the following command to deploy to Kubernetes: kubectl apply -f deployment.yaml.
Similarly, you’d need to deploy a service to provide a way to get to the running containers that make up the deployment and an ingress to allow external access to that service:
I won’t go into the details here, but the key is that you need to understand and write YAML along with understanding the YAML for deployments, services, and ingress.
If this sounds a bit cumbersome, it is, and you are not the first one to think so. Helm charts were developed to bring together a deployment, service, and ingress in a way to simplify deployments.
The basic structure of a Helm chart is as follows:
The base Helm chart has templates which you can configure to avoid having to write your own deployment, service, and ingress YAML files. If you have not already written those files, it is probably easier to use that configuration option. You can, however, just copy your files into the template directory and then deploy the application with helm install dirname, where dirname is the name of the directory in which the chart files are located.
Helm is billed as the “Package Mana ger for Helm” and does make it easier to deploy an application configured through multiple YAML files and possibly multiple components. There are, however, shortcomings, including concerns over security and relying on only static configuration versus being able to have programmatic control. Operators look to address these issues. Read more about them here: https://github.com/operator-framework.
Iterative testing
After you figure out how to deploy your application, you need to test your application. The development and test process is iterative, so you probably need to rebuild, deploy, test a number of times. Check out our article on test-driven development.
If you are testing in Docker, you need to build a new image, start a container, and then access the image during each iteration. If you are testing in Kubernetes, you have to delete pods, redeploy, and so forth. Even when using Helm charts, this will take a number of commands and time for things to spin up. The key challenge is that this all takes time and that can slow down your development process.
So far, we’ve talked about building and testing an image for your application, but not what’s in your application itself. I can’t talk about your specific application, but let’s look at what you should build into your application in order to support a cloud-native deployment.
Health checking your applications
The first thing you need to build into your application is support for liveness and readiness endpoints. Kubernetes has built-in functionality to check these endpoints:
Liveness – restart the container when the liveness endpoint does not respond indicating the application is alive.
Readiness – defer sending traffic until application is ready to accept traffic, or step sending traffic if application is no longer able to accept traffic.
Kubernetes supports three probe types.
Running a command (through shell in the container)
HTTP probe
TCP probe
As a Node.js developer, you will probably use an HTTP probe. A response code of >=200 and less than 400 means everything is okay.
You configure the probes through the deployment YAML. For example:
In this code listings, I added a liveness prob on port 3000 using the /live path.
Additionally:
initialDelaySeconds controls how quickly to start checking (in order to allow enough time for the application to start up).
periodSeconds controls how often to check.
timeoutSeconds is 1 by default, so if your probe can take longer than that you’ll have to set that value as well.
failureThreshold controls how many times the probe needs to fail before the container will restart. You want to ensure that you don’t get into a loop where the application continues to restart and never has long enough to become ready to serve the probes.
Similarly, I added a readinessProbe on port 3000 using the /ready path. To test this out, I created an application that only responds successfully to the readinessProbe after 10 seconds. If I query the pods after deploying the application the output is:
User10minikube1:~/test$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/test1 0/1 Running 0 6s
Which shows 0/1 under READY indicating that the application is not yet ready to accept traffic. After 10 seconds, it should show 1/1, and Kubernetes will start routing traffic to the pod. This is particularly important when we scale up or down as we don’t want to route traffic to a new container until it is ready or else it will slow responses even though another container is up and already capable of responding quickly.
The readiness and liveness probes may be as simple as:
However, you will often need to check the liveness and readiness of other components within your application. This can get complicated if you have a lot of subcomponents. If you have a lot of subcomponents to check, I recommend using a package like CloudNativeJS/cloud-health that allows you to register additional checks.
Logging
Node.js developers also need to know how to do logging in a cloud-native environment. In container development, writing logs out to disk does not generally make sense because of the extra steps needed to make the logs available outside the container—and they will be lost once the container is stopped.
Logging to standard out (stdout) is the Cloud native way, and structured logging (for example using JSON) is the current trend. One of my current favorite modules is pino which is a fast, structured logger and is easy to use.
Kubernetes has support for watching standard out, so it’s easy to get the log output, for example you can use kubectl logs -f <name of pod> where <name of pod> is the name of a specific pod as follows:
user1@minikube1:~/test$ kubectl get pods
NAME READY STATUS RESTARTS AGE
test1-7bbc4449b4-98w2j 1/1 Running 0 30s
test1-7bbc4449b4-zqx6f 1/1 Running 0 30s
user1@minikube1:~/test$ kubectl logs -f test1-7bbc4449b4-98w2j
{"level":30,"time":1584451728646,"pid":1,"hostname":"test1-7bbc4449b4-98w2j","msg":"/live","v":1}
{"level":30,"time":1584451734486,"pid":1,"hostname":"test1-7bbc4449b4-98w2j","msg":"ready","v":1}
{"level":30,"time":1584451738647,"pid":1,"hostname":"test1-7bbc4449b4-98w2j","msg":"/live","v":1}
{"level":30,"time":1584451748643,"pid":1,"hostname":"test1-7bbc4449b4-98w2j","msg":"/live","v":1}
{"level":30,"time":1584451758647,"pid":1,"hostname":"test1-7bbc4449b4-98w2j","msg":"/live","v":1}
{"level":30,"time":1584451768643,"pid":1,"hostname":"test1-7bbc4449b4-98w2j","msg":"/live","v":1}
{"level":50,"time":1584451768647,"pid":1,"hostname":"test1-7bbc4449b4-98w2j","msg":"down","v":1}
{"level":30,"time":1584451778648,"pid":1,"hostname":"test1-7bbc4449b4-98w2j","msg":"/live","v":1}
{"level":30,"time":1584451788643,"pid":1,"hostname":"test1-7bbc4449b4-98w2j","msg":"/live","v":1}
Cloud deployment environments like the IBM Cloud also have more sophisticated options for collecting and aggregating logs across containers, including LogDNA or the elastic stack Elasticsearch, Logstash, and Kibana.
Gathering metrics
In a Kubernetes deployment your application is running in containers, there may be multiple copies of each container, and it is not necessarily easy to find or access those containers in order to gather information about how your application is running. For this reason, it is important to export the key metrics from your container that you need to understand and track the health of your application.
Prometheus is the defacto standard on this front. It defines a set of metrics that you should export and gives you the ability to add additional application-specific metrics that are important to your business.
As an example, the following shows an example of the metrics returned using one of the available Prometheus packages:
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 1.7980069999999984 1585160007722
# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 0.931571000000001 1585160007722
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 2.7295780000000067 1585160007722
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1584451714
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 17645568 1585160007723
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 881635328 1585160007723
# HELP process_heap_bytes Process heap size in bytes.
# TYPE process_heap_bytes gauge
process_heap_bytes 90226688 1585160007723
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 20 1585160007723
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 201354
# HELP nodejs_eventloop_lag_seconds Lag of event loop in seconds.
# TYPE nodejs_eventloop_lag_seconds gauge
nodejs_eventloop_lag_seconds 0.000200848 1585160007723
# HELP nodejs_active_handles Number of active libuv handles grouped by handle type. Every handle type is C++ class name.
# TYPE nodejs_active_handles gauge
nodejs_active_handles{type="Socket"} 2 1585160007722
nodejs_active_handles{type="Server"} 1 1585160007722
# HELP nodejs_active_handles_total Total number of active handles.
# TYPE nodejs_active_handles_total gauge
nodejs_active_handles_total 3 1585160007722
# HELP nodejs_active_requests Number of active libuv requests grouped by request type. Every request type is C++ class name.
# TYPE nodejs_active_requests gauge
nodejs_active_requests{type="FSReqCallback"} 2
# HELP nodejs_active_requests_total Total number of active requests.
# TYPE nodejs_active_requests_total gauge
nodejs_active_requests_total 2 1585160007722
# HELP nodejs_heap_size_total_bytes Process heap size from node.js in bytes.
# TYPE nodejs_heap_size_total_bytes gauge
nodejs_heap_size_total_bytes 5971968 1585160007723
# HELP nodejs_heap_size_used_bytes Process heap size used from node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 4394216 1585160007723
# HELP nodejs_external_memory_bytes Nodejs external memory size in bytes.
# TYPE nodejs_external_memory_bytes gauge
nodejs_external_memory_bytes 3036410 1585160007723
# HELP nodejs_heap_space_size_total_bytes Process heap space size total from node.js in bytes.
# TYPE nodejs_heap_space_size_total_bytes gauge
nodejs_heap_space_size_total_bytes{space="read_only"} 262144 1585160007723
nodejs_heap_space_size_total_bytes{space="new"} 1048576 1585160007723
nodejs_heap_space_size_total_bytes{space="old"} 3256320 1585160007723
nodejs_heap_space_size_total_bytes{space="code"} 425984 1585160007723
nodejs_heap_space_size_total_bytes{space="map"} 528384 1585160007723
nodejs_heap_space_size_total_bytes{space="large_object"} 401408 1585160007723
nodejs_heap_space_size_total_bytes{space="code_large_object"} 49152 1585160007723
nodejs_heap_space_size_total_bytes{space="new_large_object"} 0 1585160007723
# HELP nodejs_heap_space_size_used_bytes Process heap space size used from node.js in bytes.
# TYPE nodejs_heap_space_size_used_bytes gauge
nodejs_heap_space_size_used_bytes{space="read_only"} 32296 1585160007723
nodejs_heap_space_size_used_bytes{space="new"} 419112 1585160007723
nodejs_heap_space_size_used_bytes{space="old"} 2852360 1585160007723
nodejs_heap_space_size_used_bytes{space="code"} 386368 1585160007723
nodejs_heap_space_size_used_bytes{space="map"} 308800 1585160007723
nodejs_heap_space_size_used_bytes{space="large_object"} 393272 1585160007723
nodejs_heap_space_size_used_bytes{space="code_large_object"} 3552 1585160007723
nodejs_heap_space_size_used_bytes{space="new_large_object"} 0 1585160007723
# HELP nodejs_heap_space_size_available_bytes Process heap space size available from node.js in bytes.
# TYPE nodejs_heap_space_size_available_bytes gauge
nodejs_heap_space_size_available_bytes{space="read_only"} 229576 1585160007723
nodejs_heap_space_size_available_bytes{space="new"} 628376 1585160007723
nodejs_heap_space_size_available_bytes{space="old"} 319880 1585160007723
nodejs_heap_space_size_available_bytes{space="code"} 3104 1585160007723
nodejs_heap_space_size_available_bytes{space="map"} 217168 1585160007723
nodejs_heap_space_size_available_bytes{space="large_object"} 0 1585160007723
nodejs_heap_space_size_available_bytes{space="code_large_object"} 0 1585160007723
nodejs_heap_space_size_available_bytes{space="new_large_object"} 1047488 1585160007723
# HELP nodejs_version_info Node.js version info.
# TYPE nodejs_version_info gauge
nodejs_version_info{version="v12.13.1",major="12",minor="13",patch="1"} 1
There are Node.js packages that can help you export the recommended set of metrics and the ones you need that are specific to your application. prom-client is my current favorite for integrating Prometheus metrics into a Node.js application.
Kubernetes also generates Prometheus metrics, and Kubernetes distributions or cloud platforms often make it easy to scrape your requested endpoints, aggregate them, and then graph the resulting data. Even if you install Kubernetes locally, there are Helm charts that make it easy to do this. The end result is that if you export Prometheus metrics from your application it will generally be quite easy for you or your operations teams to aggregate, chart, and alert on this data.
Maintaining and upgrading your applications
As with all application development, you have to think about maintaining and upgrading your application. Kubernetes helps you update your application deployment by rolling out updated containers, but it does not help you update the contents of the containers you built.
You have to have a plan for how to maintain and upgrade each component you use to build your container or that you add into your container. These range from docker images, Dockerfiles, live and readiness endpoints, logging, and metrics.
If you have X projects with Y containers, you can end up with a big number of container images that you must maintain. When you decide to update a dependency (for example the logger or Prometheus package etc.), you have to figure out which images need to be updated, what versions they are already using, and the like.
What happens if you move to a new team? You’ll need to figure out what their docker file look like, how the deployment artifacts are organized, what endpoint names are used for liveness and readiness, which logger they are using, and more just to get started.
I don’t know about you, but at this point I start to think “enough is enough” — I just want to focus on the application code —. Do I really need to think about Docker, Dockerfiles, YAML and all this stuff that’s outside of my application itself?
Separating concerns
What I’d really like is some separation of concerns and tooling to help me build/test efficiently using best practices agreed on by my organization. Something along these lines:
The two main components are:
a common stack agreed on by developers, architects, and operations which is re-used across projects within my organization
the application and additional packages that I need for the applications that I’m working on
For me, having consistency in the areas that don’t affect the “secret sauce” for the application I’m working on brings great benefits in ongoing maintenance and upgrades and even moving across teams. I look at this as Consistency where it makes sense as opposed to uniformity.
Tools to help with consistency
There are a few tools that can help isolate the tasks that you don’t need to worry about so that you can just focus on your code. I’ll introduce three new ones that I think can help you succeed.
Appsody
The Appsody open source project attempts to separate the operations aspects of application development to allow developers to focus on their application code, while at the same time building on a common Stack(s) that can be shared across an organization.
Appsody helps in the following ways:
Builds a base stack that you can layer applications on top of. This include most of the base components I’ve mentioned above which are needed for cloud-native deployment.
Accelerates the iterative local development and test cycle.
Simplifies creating Docker images using best practices and deploying them to Kubernetes.
There are Node.js-specific stacks in Appsody, including
nodejs – base stack for any Node.js application
nodejs-express – base stack with express built in
nodejs-loopback – base stack with loopback built in
nodejs-functions – base stack where you can write functions in the Function as a service(FaaS) style
I’m excited about Appsody offering reusable stacks that can be shared across teams, and that allows architects and operators to update the stack separately from the application itself. This is a newish approach (hence, the incubator in the path for the stacks), so I won’t be surprised if there are tweaks to the approach or tools as people experiment and get more familiar with it.
It’s great to see an open source code base starting to form and allow organizations to start proving out the approach. The existing stacks integrate liveness, readiness endpoints, and export a Prometheus endpoint. At the same time, the base Appsody framework also provides commands for building and deploying containers to Kubernetes or testing locally in Docker. The result is that much of what you need to know or understand as a Node.js developer for a coud-native deployment is supported by the tooling.
While I still think it is good for you to have a basic understanding of Kubernetes and cloud-native development, these new tools ensure you won’t need to be an expert on the ins and outs of the deployment YAML files, multistage Docker files, and the like.
Codewind
If you like what you see in Appsody but prefer a graphical user interface, you might want to check out the open source Codewind project. It adds a nice GUI, has Appsody integration, and allows you to bring your own visual editor (for example VS Code). It also adds in performance monitoring and Tekton pipelines. I’m more of a command line vi person, but if you’re interested you can check out it out here.
IBM Cloud Pack for Applications
Finally, if you are sold on this approach and want to build on a supported stack from IBM, check out the IBM Cloud Pack for Applications which includes the stack- based approach as well as support for Node.js.
Final thoughts
I hope you’ve gained a more concrete understanding of what cloud-native development and Kubernetes means to you as a Node.js developer and hope I’ve piqued your interest in stack-based development and Appsody. I talked on this subject at the most recent Node+JS Interactive conference and if you’d like to watch the recording, check out the slides and watch the recording.
This post originally appeared in Appium::Pro, Appium’s newsletter. Appium, a hosted Project at the OpenJS Foundation, is an open-source platform that enables automated testing of mobile and desktop apps on iOS, Android, Windows, Mac, and more.
Did you know that Appium is just a web server? That’s right! I’m going to write a full edition at some point on the WebDriver Protocol, which is the API that Appium implements (and which, as you can tell by the name, was invented for the purpose of browser automation). But for now, let’s revel in the knowledge that Appium is just like your backend web or API server at your company. In fact, you could host Appium on a server connected to the public internet, and give anybody the chance to run sessions on your devices! (Don’t do this, of course, unless you’re a cloud Appium vendor).
You may be used to writing Appium code in Java or some other language, that looks like this:
driver = new IOSDriver<WebElement>(new URL("http://localhost:4723/wd/hub"), capabilities);
WebElement el = driver.findElement(locator);
System.out.println(el.getText());
driver.quit();
This looks like Java code, but what’s happening under the hood is that each of these commands is actually triggering an HTTP request to the Appium server (that’s why we need to specify the location of the Appium server on the network, in the IOSDriver constructor). We could have written all the same code, for example, in Python:
driver = webdriver.Remote("http://localhost:4723/wd/hub", capabilities)
el = driver.find_element(locator)
print(el.text)
driver.quit()
In both of these cases, while the surface code looks different, the underlying HTTP requests sent to the Appium server (and the responses coming back) are the same! This is what allows Appium (and Selenium, where we stole this architecture from) to work in any programming language. All someone needs to do is code up a nice little client library for that language, that converts that language’s constructs to HTTP requests.
What all this means is that we technically don’t need a client library at all. It’s convenient to use one, absolutely. But sometimes, we want to just run an ad-hoc command against the Appium server, and creating a whole new code file and trying to remember all the appropriate syntax might be too much work. In this case, we can just use curl, which is a command line tool used for constructing HTTP requests and showing HTTP responses. Curl works on any platform, so you can download it for your environment if it’s not there already (it comes by default on Macs, for example). There are lots of options for using curl, and to use it successfully on your own, you should understand all the components of HTTP requests. But for now, let’s take a look at how we might encode the previous four commands, without any Appium client at all, just by using curl!
# 0. Check the Appium server is online
> curl http://localhost:4723/wd/hub/status
# response:
{"value":{"build":{"version":"1.17.0"}},"sessionId":null,"status":0}
# 1. Create a new session
> curl -H 'Content-type: application/json' \
-X POST \
http://localhost:4723/wd/hub/session \
-d '{"capabilities": {"alwaysMatch": {"platformName": "iOS", "platformVersion": "13.3", "browserName": "Safari", "deviceName": "iPhone 11"}}}'
# response:
{"value":{"capabilities":{"webStorageEnabled":false,"locationContextEnabled":false,"browserName":"Safari","platform":"MAC","javascriptEnabled":true,"databaseEnabled":false,"takesScreenshot":true,"networkConnectionEnabled":false,"platformName":"iOS","platformVersion":"13.3","deviceName":"iPhone 11","udid":"140472E9-8733-44FD-B8A1-CDCFF51BD071"},"sessionId":"ac3dbaf9-3b4e-43a2-9416-1a207cdf52da"}}
# save session id
> export sid="ac3dbaf9-3b4e-43a2-9416-1a207cdf52da"
Let’s break this one down line by line:
Here we invoke the curl command, passing the -H flag in order to set an HTTP request header. The header we set is the Content-type header, with value application/json. This is so the Appium server knows we are sending a JSON string as the body of the request. Why do we need to send a body? Because we have to tell Appium what we want to automate (our “capabilities”)!
-X POST tells curl we want to make a POST request. We’re making a POST request because the WebDriver spec defines the new session creation command in a way which expects a POST request.
We need to include our URL, which in this case is the base URL of the Appium server, plus /session because that is the route defined for creating a new session.
Finally, we need to include our capabilities. This is achieved by specifying a POST body with the -d flag. Then, we wrap up our capabilities as a JSON object inside of an alwaysMatch and a capabilities key.
Running this command, I see my simulator pop up and a session launch with Safari. (Did the session go away before you have time to do anything else? Then make sure you set the newCommandTimeout capability to 0). We also get a bunch of output like in the block above. This is the result of the new session command. The thing I care most about here is the sessionId value of ac3dbaf9-3b4e-43a2-9416-1a207cdf52da, because I will need this to make future requests! Remember that HTTP requests are stateless, so for us to keep sending automation commands to the correct device, we need to include the session ID for subsequent commands, so that Appium knows where to direct each command. To save it, I can just export it as the $sid shell variable.
Now, let’s find an element! There’s just one element in Appium’s little Safari welcome page, so we can find it by its tag name:
# 2. Find an element
> curl -H 'Content-type: application/json' \
-X POST http://localhost:4723/wd/hub/session/$sid/element \
-d '{"using": "tag name", "value": "h1"}'
# response:
{"value":{"element-6066-11e4-a52e-4f735466cecf":"5000","ELEMENT":"5000"}}
# save element id:
> export eid="5000"
In the curl command above, we’re making another POST request, but this time to /wd/hub/session/$sid/element. Note the use of the $sid variable here, so that we can target the running session. This route is the one we need to hit in order to find an element. When finding an element with Appium, two parameters are required: a locator strategy (in our case, “tag name”) and a selector (in our case, “h1”). The API is designed such that the locator strategy parameter is called using and the selector parameter is called value, so that is what we have to include in the JSON body.
The response we get back is itself a JSON object, whose value consists of two keys. The reason there are two keys here is a bit complicated, but what matters is that they each convey the same information, namely the ID of the element which was just found by our search (5000). Just like we did with the session ID, we can store the element ID for use in future commands. Speaking of future commands, let’s get the text of this element!
# 3. Get text of an element
> curl http://localhost:4723/wd/hub/session/$sid/element/$eid/text
# response:
{"value":"Let's browse!"}
This curl command is quite a bit simpler, because retrieving the text of an element is a GET command to the endpoint /session/$sid/element/$eid/text, and we don’t need any additional parameters. Notice how here we are using both the session ID and the element ID, so that Appium knows which session and which element we’re referring to (because, again, we might have multiple sessions running, or multiple elements that we’ve found in a particular session). The response value is the text of the element, which is exactly what we were hoping to find! Now all that’s left is to clean up our session:
This last command can use all the default curl arguments except we need to specify that we are making a DELETE request, since that is what the WebDriver protocol requires for ending a session. We make this request to the endpoint /session/$sid, which includes our session ID so Appium knows which session to shut down.
That’s it! I hope you’ve enjoyed learning how to achieve some “low level” HTTP-based control over your Appium (and Selenium) servers!
This blog post was written by John Walicki, CTO for Edge/IoT Advocacy in the Developer Ecosystem Group of IBM Cognitive Applications Group and originally published on IBM Developer.
Learn how to build a weather dashboard using a personal weather station, Node-RED, Weather Underground, and The Weather Company APIs and the node-red-contrib-twc-weather nodes. This tutorial demonstrates how to display hyper-local weather information from a residential or farming weather station.
Learning objectives
In this tutorial, you will:
Learn the basics of personal weather stations (PWS)
Connect your PWS to Weather Underground (WU) and view PWS data on WU
Register for a The Weather Company (TWC) API key
Get started with the TWC API documentation
Learn about Node-RED (local and on IBM Cloud)
Explore the node-red-contrib-twc-weather Node-RED PWS node examples
Import / Deploy the Weather Dashboard example
Display PWS data in your Weather Dashboard
Build a Severe Weather Alert Map Node-RED Dashboard using TWC APIs
Build a Call for Code Water Sustainability solution
If you don’t have a PWS, you can still get a time-restricted TWC API key by joining Call for Code (which gives you access to most of the TWC PWS APIs)
Estimated time
Completing this tutorial should take about 30 minutes.
Steps
Introduction to personal weather stations
Wikipedia defines a personal weather station as a set of weather measuring instruments operated by a private individual, club, association, or business (where obtaining and distributing weather data is not a part of the entity’s business operation). Personal weather stations have become more advanced and can include many different sensors to measure weather conditions. These sensors can vary between models but most measure wind speed, wind direction, outdoor and indoor temperatures, outdoor and indoor humidity, barometric pressure, rainfall, and UV or solar radiation. Other available sensors can measure soil moisture, soil temperature, and leaf wetness.
The cost of a sufficiently-accurate personal weather station is less than $200 USD; they have become affordable for citizen scientists and weather buffs.
Connect your PWS to Weather Underground
Many PWS brands offer the ability to connect and send weather data to cloud based services. Weather Underground, a part of The Weather Company, an IBM Business, encourages members to register their PWS and send data to http://www.wunderground.com.
Members can view their personal weather station data on Weather Underground
Get a TWC API key and get started with the TWC API documentation
In addition to the wunderground.com dashboard, the PWS data is available through your API Key and a set of robust TWC Restful APIs. Copy your API Key and click on the View API Documentation button.
Register for a TWC API key
If you don’t have a Personal Weather Station, you can still register for a time-restricted TWC API key by joining Call for Code 2020. The API Key is valid from March 1 to October 15, 2020. This API key gives you access to most of the TWC Personal Weather Station APIs. You can complete this tutorial using this API key.
Learn about Node-RED
Node-RED is an open source programming tool for wiring together hardware devices, APIs, and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.
The node-red-contrib-twc-weather GitHub repository includes an example flow that exercises each of the Node-RED PWS APIs. You can learn about the nodes and their configuration options by clicking on each node and reading its comprehensive node information tab. Import this PWS-Examples.json flow into your Node-RED Editor and Deploy the flow. Don’t forget to paste in your TWC PWS API key. If you want to explore personal weather station data but don’t have your own PWS, you can query the weather station data at the Ridgewood Fire Headquarters using the StationID KNJRIDGE9
Import / Deploy the weather dashboard example
Now that the Node-RED node-red-contrib-twc-weather nodes are able to query weather data, let’s build an example Weather Node-RED Dashboard that displays Personal Weather Station current and historical data on a map, in a table, a gauge, and on a chart. The PWS API key includes access to the TWC 5 Day Forecast, which is displayed with weather-lite icons. This flow requires node-red-dashboard, node-red-node-ui-table, and node-red-contrib-web-worldmap. Import this PWS-Dashboard.json flow and Deploy the flow.
Display PWS data in your weather dashboard
Launch the Node-RED Dashboard and experiment with the current conditions, forecast, and map. The Call for Code TWC API key might not have access to private PWS historical data.
Build a Severe Weather Alert Map Node-RED Dashboard using TWC APIs
In addition to the node-red-contrib-twc-weather Node-RED nodes, you can review the TWC Severe Weather API Documentation and use the http request node and your API Key to make calls directly.
Build a Call for Code Water Sustainability solution!
Now that you have completed this tutorial, you are ready to modify these example flows and Node-RED Dashboard to build a Call for Code Water Sustainability solution.
