Projects

Introducing kepler.gl 3.0


Version 3.0 of kepler.gl is now available, bringing new geospatial features to kepler.gl application users and dramatic improvements to developers using the kepler.gl libraries.

By Ib Green, Foursquare and Shan He, Foursquare

What’s New in the kepler.gl 3.0 Application?

Post-Processing Effects

The post-processing effects feature from Foursquare Studio has been upstreamed into the open source kepler.gl, bringing lights and shadows, and a library of image processing effects.

Lights and shadows, depth-of-field and other effects in play.

For more information, see Elevate Your Maps with Foursquare’s Artistic Geospatial Tools, which describes the feature set in Foursquare Studio that has now been merged back to kepler.gl.

Apache Arrow brings speed and memory usage improvements

kepler.gl 3.0 starts adding support for the Apache Arrow format, enabling dramatic increases in loading performance and maximum dataset sizes. 

In this release, kepler.gl can load GeoArrow files, which can be visualized with the PolygonLayer. See the roadmap section below for more general Apache Arrow file support and support for other layers.

10x faster! Loading 1 million polygons takes only ~2 seconds with GeoArrow format
vs. ~20 seconds with GeoJson format.

Incremental loading of 11 million row GeoArrow dataset

The Apache Arrow tables support all features, including filtering and tabular viewing.

The trick behind the massive performance boost is that Apache Arrow files are loaded as binary data. Binary data loads

  • avoid time-consuming parsing that is required for textual formats like GeoJSON
  • allow kepler.gl to skip some costly data conversion steps when preparing the data for rendering on the GPU

Binary data support in kepler.gl has been a long time in the works and required deep refactoring of the kepler.gl core logic. For some additional information, see the DataContainer bullet in the kepler.gl library updates below and the roadmap.

Substantial parts of the GeoArrow support was contributed by the Center for Spatial Data Science at the University of Chicago.

Foursquare integration

A new Foursquare cloud plugin is available in kepler.gl, letting users save, load and publish kepler.gl maps via their Foursquare accounts. It also enables users to open kepler maps in Foursquare Studio, from which additional functions such as sharing, web publishing, and much more is available.

Foursquare is building a full suite of freemium tools on top of kepler.gl, and kepler.gl can now be used as a part of the Foursquare Studio platform through this new integration.

Foursquare accounts are free to create, offer a gigabyte of free cloud storage, and give access to a wide range of additional geospatial tools.

The Foursquare cloud plugin is an addition to the existing CARTO and Dropbox plugins, giving the user a wider range of options for how to store kepler.gl maps.  

Naturally, cloud plugins are optional to use, and none of your data is shared with any cloud provider without the user’s explicit login and save action.

Maplibre Base Map Renderer

The default base map renderer has been changed from Mapbox to Maplibre to ensure that kepler.gl is using an open source basemap that is actively supported by a strong community.
The maplibre integration was provided by Birk Skyum.

Minor features

Apart from the headline features listed above, users may want to be on the lookout for numerous small performance and usability improvements in areas such as:

  • Animations
  • Map styles
  • Dataset handling
  • Viewports
  • Layers
  • Tooltips

What’s new for developers in kepler.gl 3.0?

The vast majority of kepler.gl 3.0 development work has taken place under the hood, improving the kepler.gl frameworks that power all kepler.gl based applications.

kepler.gl is a framework that lets developers quickly build their own customized tools for visual geospatial analytics. In fact, the popular geospatial analytics application hosted at https://kepler.gl is simply a small demo application built on top of the kepler.gl framework. 

TypeScript

The entire kepler.gl code base has been converted to TypeScript. This helps mitigate code base complexity and has increased developer velocity:

  • Type checks catch mistakes and provide a safety net when making changes to the code.
  • Developers can inspect type definitions to confirm expected data formats are.

The TypeScript migration was provided by Foursquare in collaboration with Action Engine.

Modules

The kepler.gl libraries are published as a suite of namespaced npm modules. This makes kepler.gl more modular and composable, and enforces stricter separation of functionality, which should help both developers and maintainers.

The modularization of kepler.gl was implemented by Foursquare in collaboration with Action Engine.

DataContainer

A new DataContainer interface encapsulates access to data tables. Before the DataContainer was introduced, all data loaded by kepler had to be converted into a common table format defined by JavaScript arrays and objects.

The DataInterface lets developers change how data tables are stored and organized internally, with having to modify all the places where.kepler.gl accesses table data. 
The new DataInterface is not just an abstract capability. It is used by the new GeoArrow loader, which can now store the loaded binary tables in a compact and efficient binary form in memory instead of having to convert them to a traditional, bulkier, non-binary JavaScript format. 

The data container was developed by Foursquare.

Customizability

Many additional React UI components now export “factories” making them overridable.

When building applications on top of the kepler.gl library, it is often desirable to override components and behaviors of kepler with custom implementations. 

Dependency Upgrades

Kepler.gl 3.0 also upgrades a number of its dependencies to the latest versions. This brings in a wide range of bug fixes and performance improvements. Examples of upgraded dependencies are:

  • deck.gl 8.9 
  • loaders.gl 4.0
  • React 18 
  • Node 18

Where is kepler.gl headed?

kepler.gl is under open governance in the OpenJS foundation, and for the last few years kepler.gl has been maintained by Foursquare, where the original creators are using kepler as an open core for building location platform products such as Foursquare Studio. 

While it took a long time for kepler.gl 3.0 to be released, development has never stopped. More than 800 commits have been made to the kepler.gl master branch after the release of kepler.gl v2.0 in February 2020. Though admittedly most of these commits have been focused on strengthening the foundations of kepler.gl rather than new features.

The current goal of the kepler.gl team is to increase release cadence, ideally to a quarterly release and also increase the amount of end-user facing geospatial features in each release.

If you’d like to participate in kepler.gl discussions or if you would consider contributing in any way, it is easy to engage with the kepler team through the Open JS Foundation at https://www.openvisualization.org/.

Roadmap

As mentioned above, the kepler.gl developer team would like to ship quarterly releases. Some of the major items we hope to work on for those release are:

Binary Data (kepler.gl 3.x)

Adding Apache Arrow support to all layers (beyond the PolygonLayer), and extending binary data support to other file formats, so that users can benefit from massive performance gains in more situations.

Cloud Native Geospatial File Formats (kepler.gl 3.x)

An exciting trend in the geospatial industry is the momentum around cloud-native geospatial file formats. We’d like to keep adding support for loading from and exporting to these formats, including GeoParquet, FlatGeobuf, PMTiles, etc.

WebGPU (kepler.gl 4.0)

Kepler.gl builds on deck.gl, and deck.gl v9 development is focused on WebGPU support. Upgrading kepler.gl to deck.gl v9 once it becomes available will bring opportunities to build even more performant and capable GPU visualizations and compute capabilities.


How to Engage