Link Search Menu Expand Document
NHM DEV DOCS
Tutorials
This site uses Just the Docs, a documentation theme for Jekyll.

LAU fossil map

Authored by: Daniel Markbreiter
Table of Contents

Example of the fossil map as it looked in LA Underwater

The LAU fossil map was developed as a way for users to explore NHM’s Southern Californian fossil specimens geospatially. Users can select counties, regions and neighborhoods and see information about of photos of fossil specimens found in the area.

Currently, there is a version for use in the in-gallery exhibit and another for the online use. Both versions are versioned in the same git repository. With the exception of a search bar functionality, the two version have the same features.

Tech stack

The fossil map’s front end is primairly built using the ArcGIS API for JavaScript 4.x, developed by Esri. Backend scripts to udpate the MongoDB use the ArcGIS API for Python in order to pull data stored in ArcGIS Online.

Tech Stack  
Operating System Ubuntu (Dokku)
Server Ngnix
Language Python 3.x
Framework Flask
Database MongoDB

Data sources

lauVectorTileLayer

Used as a base layer in the fossil map application. Includes polygons of counties, regions and neighborhoods, packaged as a vector tile layer and stored on ArcGIS Online. Counties, regions and neighborhoods toggle visibility based on zoom level, which was set at the time of vector tile packaging.

Below is a list of data sources for the counties, regions and neighborhoods. All polygon layers were converted into geojsons and simplified using mapshaper’s Simplify tool with the “prevent shape removal” option toggled and the default “Visvalingam / weighted area”. Using the -clean command and the allow-empty flag used, gaps were filled in each of the layers. These geojsons were imported into ArcGIS Pro and bundled as a vector tile layer that was exported to ArcGIS Online as lauVectorTileLayer, which is used as the base tile in the fossil map.

Each of the individual geojson files were exported to ArcGIS Online as hosted feature layers. View layers were generated from these feature layers, used in a previous version of the exhibit app.

Labeling features

Polygon Labels for the vector tile layer are visualized using centroids of the polygons and read into the app as a feature layer.

While vector tile layers support labeling, they do not offer support for the ArcGIS lableing engine that detects collision in the ArcGIS API for JavaScript. The use of centroids offer a way to bring labels that can reduce label collisions for the cost of three small requests to ArcGIS Online.

  • lauCountiesCentroids

    • Centroids generated from the lauCounties feature layer in ArcGIS Pro and read into the fossil map as a feature layer for labeling

  • lauRegionsCentroids

    • Centroids generated from the lauRegions feature layer in ArcGIS Pro and read into the fossil map as a feature layer for labeling

  • lauNeighborhoodsCentroids

    • Centroids generated from the lauNeighborhoods feature layer in ArcGIS Pro and read into the fossil map as a feature layer for labeling

Localities Layer

The visible point features on the map are generated localities from Invetebrate Paleontology (IP) and Vertebrate Paleontology (VP).

IP localities were exported from the Sites module of IP’s EMu, the collection database used by R&C. The Sites module does not include the required taxonomic data used by the map. Associated Catalogue records (i.e. specimen records) were returned by searching for attached objects, since locality (i.e. Sites) records are attached to catalog records.

VP localities were produced from collection manager Sam McLeod.

The localities layer visualized in the map are hosted as feature layer on ArcGIS Online as a layer view.

lauAllAreasFinalCopy.geojson

A geojson layer of all the polygon features is used for the update.py to reference when performing spatial queries to populate the MongoDB. This geojson was created because of inconsistencies between how ArcGIS FeatureLayers format geospatial data and how MongoDB formats them (relying on the geojson specification). Specifically, multipart polygons must have self-closing polygons, whereas for ArcGIS FeatureLayers they do not.

Therefore, it was easier to clean the data from the lauAreas feature layer and export as a geojson layer to be used by the update.py to query against. Since the areas used by the app are quite static (it was deemed unlikely for the addition or modification of areas to occur), this solution was considered acceptable and a more predictable outcome.

Photos

Photos of fossils and other specimens are stored in a Digital Ocean Spaces (i.e. object storage). References to those URLs and the accompanying geospatial/metadata are stored in a ArcGIS Online hosted feature layer named “lauPhotos”.

When the update.py script is run, it pulls the data from the lauPhotos layer and queries the point features against the areas from lauAllareasFinalCopy.geojson.

MongoDB

The database is used mainly as a cacheing layer. Queries for the map could be made enitrely using the ArcGIS API for JavaScript’s Query calls, but the spatial queries took considerable time to return features.

Instead, an intersection query of each polygon is performed against the localities hosted feature. The results of the query are stored as objects in the Area collection in MongoDB.

Because of formatting concerns, a derivative geojson of all areas (i.e. all polygons used in the vector tile layer) was created. Specifically, the geojson has polygons with rings that self closed, something that is allowed in ArcGIS features but not for MongoDB features.

Update script

A cron job on the Dokku production server runs the update.py script on a daily basis. This makes sure that any changes made to the lauLocalities and lauPhotos hosted feature layers are reflected in the map.

The update.py script uses the ArcGIS API for Python to see if changes have been made to the two hosted feature layers. If changes have been made, the script will iterate through features and update the mongoDB document by document.

Installation

  1. Clone the github repository

    git clone https://github.com/naturalhistorymuseumofla/lau-interactive.git

  2. Switch to dokku branch

    The dokku branch is the branch that is deployed to the dokku production server on the dokku-production Digital Ocean droplet.

    git switch dokku

  3. Check if python 3.x is installed

    Enter python -v inter terminal. If the response is Python 2.7.16 or some other version of python 2.x, than you will need to install python 3.

    Do not use the system Python installation on macOS!

    As of macOS 12.3, python 2.7 was removed from macOS. Any previous versions of macOS include Python 2.

  4. Create a virtual environment

    While not strictly required, it is suggested that a virtual environment is created for the project, so as to not pollute yoru global installation of Python. venv is a simple virtual environment manager and module, whereas anaconda and its bundled conda is a more robust option.

  5. Install from requirements.txt

    Running the following command inside the base of the project directory will install all the required dependencies for the project.

    pip install -r requir÷ements.txt

  6. Install MongoDB and start the service

    Installing MongoDB should be done via homebrew using the following commands. It is suggested, however, that you consult the official documentation on Installing MongoDB Community Edition on macOS

    1. Tap the MongoDB Homebrew Tap to download the official Homebrew formula for MongoDB and the Database Tools, by running the following command in your macOS Terminal: 
      brew tap mongodb/brew
    2. To update Homebrew and all existing formulae: 
      brew update
    3. To install MongoDB, run the following command in your macOS Terminal application (version 5.0 is the latest version as of the time of this documents writing): 
      brew install mongodb-community@5.0

    4. Start the MongoDB service:

      brew services start mongodb-community@5.0

  7. Add .env file

    Environment variables are used in the app for API keys and for ArcGIS Online/DB credentials. Please ask a developer for these variables, or check the environment variables for the fossilmap app on the dokku-production server.

  8. Run the app

    flask run

    This should initiate the update.py script to pull data from ArcGIS Online and populate the MongoDB. After a successful execution of update.py, the app should start.

Table of Contents