README.md 5.7 KB
Newer Older
1
# WSM
2

Luís de Sousa's avatar
Luís de Sousa committed
3
World Soil Museum - REST API
4

5
## App structure
6
7
8
9
10
11
12
13
14
15
16
17
18
19

REST APP is on [appWSM.py](./appWSM.py) and then calling modules like `resources`. This app is based on:

- Flask
- Flask_restfull

App requires access to ISRIC database to retrieve profile information.

This app was build before the openapi standard and generic request and information can be found at the URL root level: `https://rest-wsm.isric.org/` Please consult the URL for the following examples:

- GET monolith pictures
- GET content filter (WRB classes, country)
- GET limit number of requested pictures

20
21
22
23
24
25
26
27
28
29
30
## REST testing

REST is has two environments:

- [https://rest-wsm.containers.wurnet.nl](https://rest-wsm.containers.wurnet.nl) (dev)
- [https://rest-wsm.isric.org](https://rest-wsm.containers.wurnet.nl) (prod)

**NOTE:** The dev env only has a limit number of available images, to prevent to much data on dev. To test the REST image and DB acccess we can use the following links:

Testing links for development:

31
32
33
34
35

- https://rest-wsm.containers.wurnet.nl/monoliths/
- https://rest-wsm.containers.wurnet.nl/monoliths/original/tif/TH-008-def.tif
- https://rest-wsm.containers.wurnet.nl/monoliths/original/tif?country=TH
- https://rest-wsm.containers.wurnet.nl/monoliths/1per/tif?classification=andosols&extra&limit=10&country=NZ,FR
36
37
38
39
40

Testing links for production:

```bash
https://rest-wsm.isric.org/monoliths/
41
https://rest-wsm.isric.org/monoliths/original/tif/CA-006-def.tif
42
https://rest-wsm.isric.org/monoliths/original/tif?country=PT
43
https://rest-wsm.isric.org/monoliths/1per/tif?classification=andosols&extra&limit=10&country=NZ,FR
44
45
```

46
47
## Database

Mendes de Jesus, Jorge's avatar
Mendes de Jesus, Jorge committed
48
49
50
App accesses ISRIC database on `scomp1270.wurnet.nl`.

**NOTE:** To copy files check section below  [Monolith picture location](#monolith-picture-location)
51

52
### DB access configuration / Data path
53

54
DB options and monolith path location is picked up from env variables on script: [common/secret.py](./common/secret.py).
55

56
57
58
59
60
61
62
63
```bash
DB_HOST=scomp1270.wurnet.nl
DB_NAME=isric
DB_USER=hass005
DB_PASSWORD=<removed>
DB_PORT=5479
MONOLITH_PATH=/data/monolith
```
64
65
66
67

### DB queries

 In the code we have the following queries:
68

69
70
71
72
| code path         | method/function        | SQL                          |
|:-----------------:|:---------------------:|:-----------------------------:|
| common/filters.py | filter_classification()|select * from web_isis.vw_isis_profiles_with_classification_geo_img_rest where classification ilike %s"|
| common/filters.py | filter_classification() |select * from web_isis.vw_isis_profiles_with_classification_geo_img_rest where classification ilike %s|
Luís de Sousa's avatar
Luís de Sousa committed
73

Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
74
**NOTE:** Views need to be refreshed when new data is added to the upstream tables
Luís de Sousa's avatar
Luís de Sousa committed
75

76
### Monolith picture location
77
78
79
80
81

Monoliths used by REST are stored on ISRIC data volume, the access to this volume is very restricted and based on IP. Rserver (scomp1457.wurnet.nl), on this server users haas005, turku001 and ruipe001 have a symbolic link to the monoliths.

There the monolith pictures can be updated or modified

82
### Monolith file copy
Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
83
84
85

On the Rserver is it better to copy the monoliths from workspace scratch into `ISRIC_Monoliths`. On the home folder of user the command would look like:

86
```bash
Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
87
88
89
turdu001@scomp1457:~$ cp -rv ./ISRIC_Workspace/scratch/Upload_ToRest/combined_processed/* ./ISRIC_Monoliths/ 
```

Mendes de Jesus, Jorge's avatar
Mendes de Jesus, Jorge committed
90
91
The FB-IT mounts are not reliable, check if "ISRIC_Monolith" is accessible, it not mount it:

92
```bash
Mendes de Jesus, Jorge's avatar
Mendes de Jesus, Jorge committed
93
94
95
96
# mount all volumes from /etc/fstab
sudo mount -a
```

Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
97
98
In case of batch file permission it is advisable (NOTE: Only sudo can do it):

99
```bash
Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
100
101
102
103
find /mnt/ISRIC_Monoliths -type d -exec chmod 777 {} \; #for directories
find /mnt/ISRIC_Monoliths -type f -exec chmod 666 {} \; #for files
```

104
## Monolith image update and implementation
105
106
107
108
109
110
111
112
113
114
115

To update monolith images, it is necessary:

- Process images to have the different sizes
- Add them to the ISRIC data volume
- Update DB (web_isis.profiles_with_rest_images?) and views

It is advisable to use a rsync software when moving files around.

REST API doesn't make any assumptions on files and will crawl the fold structure for the pictures. It maybe necessary some code changes if there are to many changes.

116
## Local Deployment
117
118

This API can be deployed using a docker container. The code resides in the
Luís de Sousa's avatar
Luís de Sousa committed
119
host system, together with the monoliths images to be served. These images
120
must be accessible in a folder named `monoliths` at the folder root (same level
121
as this file). This code folder is then mapped into the container at startup.
Luís de Sousa's avatar
Luís de Sousa committed
122
123
124
125

The Dockerfile present in this project sets up the API with Gunicorn and nginx.
To build the container image change to the project folder and run:

126
127
128
`docker build -t wsm.rest .`

The resulting image is named `wsm/rest`.
Luís de Sousa's avatar
Luís de Sousa committed
129

130
To run the image it is necessary to map the right environment variables to connect to the database as well as mount the source code folder and bind port 8000 (where gunicorn listens):
Luís de Sousa's avatar
Luís de Sousa committed
131

132
Docker run command with envfile:
133

134
```bash
135
136
137
138
docker run -dit \
   --env-file /path/to/envfile \
   -v /data/monoliths:/data/monoliths \
   --name wsm.rest -p 8000:8000 wsm.rest
139
```
140

141
Docker run command with variables inline:
142

143
```bash
144
145
146
147
148
149
150
151
152
docker run -dit \
    -e DB_HOST=scomp1270.wurnet.nl \
    -e DB_NAME=isric \ 
    -e DB_USER=haas005 \
    -e DB_PASSWORD=<REMOVED>
    -e MONOLITH_PATH=/data/monolith 
    -e DB_PORT=5479 \
    -v /data/monoliths:/data/monoliths 
    --name wsm.rest -p 8000:8000 wsm.rest
153
```
Luís de Sousa's avatar
Luís de Sousa committed
154

155
Your container will now be available on [localhost:8000](http://localhost:8000)
156

157
## K8s deployment
158

159
- `ENV` variables are set as k8s secrets on the [ISRIC deployment repository](https://git.wur.nl/isric/ict/k8s-deployments)
160

161
- Application follows the standard k8s deployment.
162

163
- TODO: Write about semanatic tagging
164

165
## Image processing and metadata
166

Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
167
Please consult the specific folder **[README](imageProcessing/README.md)**