README.md

# WSM

World Soil Museum - REST API

## App structure

REST APP is on [appWSM.py](./appWSM.py) and then calling modules like `resources`. This app is based on:

- Flask
- Flask_restfull

App requires access to ISRIC database to retrieve profile information.

This app was build before the openapi standard and generic request and information can be found at the URL root level: `https://rest-wsm.isric.org/` Please consult the URL for the following examples:

- GET monolith pictures
- GET content filter (WRB classes, country)
- GET limit number of requested pictures

## REST testing

REST is has two environments:

- [https://rest-wsm.containers.wurnet.nl](https://rest-wsm.containers.wurnet.nl) (dev)
- [https://rest-wsm.isric.org](https://rest-wsm.containers.wurnet.nl) (prod)

**NOTE:** The dev env only has a limit number of available images, to prevent to much data on dev. To test the REST image and DB acccess we can use the following links:

Testing links for development:


- https://rest-wsm.containers.wurnet.nl/monoliths/
- https://rest-wsm.containers.wurnet.nl/monoliths/original/tif/TH-008-def.tif
- https://rest-wsm.containers.wurnet.nl/monoliths/original/tif?country=TH
- https://rest-wsm.containers.wurnet.nl/monoliths/1per/tif?classification=andosols&extra&limit=10&country=NZ,FR

Testing links for production:

```bash
https://rest-wsm.isric.org/monoliths/
https://rest-wsm.isric.org/monoliths/original/tif/CA-006-def.tif
https://rest-wsm.isric.org/monoliths/original/tif?country=PT
https://rest-wsm.isric.org/monoliths/1per/tif?classification=andosols&extra&limit=10&country=NZ,FR
```

## Database

App accesses ISRIC database on `scomp1270.wurnet.nl`.

**NOTE:** To copy files check section below  [Monolith picture location](#monolith-picture-location)

### DB access configuration / Data path

DB options and monolith path location is picked up from env variables on script: [common/secret.py](./common/secret.py).

```bash
DB_HOST=scomp1270.wurnet.nl
DB_NAME=isric
DB_USER=hass005
DB_PASSWORD=<removed>
DB_PORT=5479
MONOLITH_PATH=/data/monolith
```

### DB queries

 In the code we have the following queries:

| code path         | method/function        | SQL                          |
|:-----------------:|:---------------------:|:-----------------------------:|
| common/filters.py | filter_classification()|select * from web_isis.vw_isis_profiles_with_classification_geo_img_rest where classification ilike %s"|
| common/filters.py | filter_classification() |select * from web_isis.vw_isis_profiles_with_classification_geo_img_rest where classification ilike %s|

**NOTE:** Views need to be refreshed when new data is added to the upstream tables

### Monolith picture location

Monoliths used by REST are stored on ISRIC data volume, the access to this volume is very restricted and based on IP. Rserver (scomp1457.wurnet.nl), on this server users haas005, turku001 and ruipe001 have a symbolic link to the monoliths.

There the monolith pictures can be updated or modified

### Monolith file copy

On the Rserver is it better to copy the monoliths from workspace scratch into `ISRIC_Monoliths`. On the home folder of user the command would look like:

```bash
turdu001@scomp1457:~$ cp -rv ./ISRIC_Workspace/scratch/Upload_ToRest/combined_processed/* ./ISRIC_Monoliths/ 
```

The FB-IT mounts are not reliable, check if "ISRIC_Monolith" is accessible, it not mount it:

```bash
# mount all volumes from /etc/fstab
sudo mount -a
```

In case of batch file permission it is advisable (NOTE: Only sudo can do it):

```bash
find /mnt/ISRIC_Monoliths -type d -exec chmod 777 {} \; #for directories
find /mnt/ISRIC_Monoliths -type f -exec chmod 666 {} \; #for files
```

## Monolith image update and implementation

To update monolith images, it is necessary:

- Process images to have the different sizes
- Add them to the ISRIC data volume
- Update DB (web_isis.profiles_with_rest_images?) and views

It is advisable to use a rsync software when moving files around.

REST API doesn't make any assumptions on files and will crawl the fold structure for the pictures. It maybe necessary some code changes if there are to many changes.

## Local Deployment

This API can be deployed using a docker container. The code resides in the
host system, together with the monoliths images to be served. These images
must be accessible in a folder named `monoliths` at the folder root (same level
as this file). This code folder is then mapped into the container at startup.

The Dockerfile present in this project sets up the API with Gunicorn and nginx.
To build the container image change to the project folder and run:

`docker build -t wsm.rest .`

The resulting image is named `wsm/rest`.

To run the image it is necessary to map the right environment variables to connect to the database as well as mount the source code folder and bind port 8000 (where gunicorn listens):

Docker run command with envfile:

```bash
docker run -dit \
   --env-file /path/to/envfile \
   -v /data/monoliths:/data/monoliths \
   --name wsm.rest -p 8000:8000 wsm.rest
```

Docker run command with variables inline:

```bash
docker run -dit \
    -e DB_HOST=scomp1270.wurnet.nl \
    -e DB_NAME=isric \ 
    -e DB_USER=haas005 \
    -e DB_PASSWORD=<REMOVED>
    -e MONOLITH_PATH=/data/monolith 
    -e DB_PORT=5479 \
    -v /data/monoliths:/data/monoliths 
    --name wsm.rest -p 8000:8000 wsm.rest
```

Your container will now be available on [localhost:8000](http://localhost:8000)

## K8s deployment

- `ENV` variables are set as k8s secrets on the [ISRIC deployment repository](https://git.wur.nl/isric/ict/k8s-deployments)

- Application follows the standard k8s deployment.

- TODO: Write about semanatic tagging

## Image processing and metadata

Please consult the specific folder **[README](imageProcessing/README.md)**