Commit d3a5c206 authored by Jorge S. Mendes de Jesus's avatar Jorge S. Mendes de Jesus
Browse files

missing transformation script and README

parent b6f900bc
"""
From a specific folder containing tiff original files will open the file and 10per,
20_200 1per 40_40050per, 80_800, original
Notice: Please check the limits on: /etc/ImageMagick-6/policy.xml Using env variables is not working
"""
path_str = """
├── 10per
│   ├── jpg
│   ├── png
│   └── tif
├── 1per
│   ├── jpg
│   ├── png
│   └── tif
├── 40_400
│   ├── jpg
│   ├── png
│   └── tif
└── original
├── jpg
├── png
└── tif
"""
import os
import glob
import subprocess
import shlex
import click
@click.command()
@click.option('--make_path', is_flag = True, default = True ,help = "Will generate outputs path on destination folde")
@click.option('--in_path',nargs=1, type=click.Path(exists=True), required=True, help = 'Path with original tiff images')
@click.option('--out_path', nargs=1, type=click.Path(exists=True), required=True,
help = f'Output path for transfomed images path structure: {path_str}')
def main(make_path, in_path, out_path):
#def main(make_path = True, in_path = "/home/jesus/git/wsm.rest/imageProcessing/tmp3/original/tif",
# out_path="/home/jesus/git/wsm.rest/imageProcessing/final"):
formats = ["jpg","png","tif"]
folders = ["10per","1per","20_200","40_400","50per","80_800","original"]
if make_path:
for folder in folders:
for format in formats:
dir_path = os.path.join(out_path, folder, format)
if not os.path.exists(dir_path):
os.makedirs(dir_path)
images = glob.glob(os.path.join(in_path,"*.tif"))
for image in images:
file_name = os.path.splitext(os.path.basename(image)) #('GB-001-def', '.tif')
file_name = file_name[0]
for folder in folders:
for format in formats:
dir_path = os.path.join(out_path, folder, format)
if folder == "original":
format_name = ""
else:
format_name = f".{folder}"
out_file_name = os.path.join(dir_path, file_name + format_name + f".{format}") #NG-006-def.50per.png
#Got problems with
if "per" in folder:
per_value = folder.split("per")[0]
command = f"convert -delete 1--1 {image} -resize {per_value}% {out_file_name}"
if "_" in folder:
res_size=folder.replace("_","x")
command = f"convert -delete 1--1 {image} -resize {res_size} {out_file_name}"
if "original" in folder:
command = f"convert -delete 1--1 {image} {out_file_name}"
print(f"running: {command}")
subprocess.call(command, shell=True)
print("all done")
if __name__ == '__main__':
main()
\ No newline at end of file
Image processing information
============================
This documentation concerns the image manipulation of the tiff images generated by the museum.
Conversion: script Orginal2ReizeImage.py
----------------------------------------
**LET OP** Conversion only after the picture have metadata
The original images are stored in folder monoliths/original/tif. The originals are converted into jpg,png and tif based on the following properties: "10per", "1per", "20_200", "40_400", "50per", "original" bellow we have the folder structure
```
.
├── 10per
│   ├── jpg
│   ├── png
│   └── tif
├── 1per
│   ├── jpg
│   ├── png
│   └── tif
├── 20_200
│   ├── jpg
│   ├── png
│   └── tif
├── 40_400
│   ├── jpg
│   ├── png
│   └── tif
├── 50per
│   ├── jpg
│   ├── png
│   └── tif
├── 80_800
│   ├── jpg
│   ├── png
│   └── tif
└── original
├── jpg
├── png
└── tif
```
Conversion is done using `magick library` that can be used on the command line, for example
```
convert -delete 1--1 ./original/tif/GB-006-def.tif -resize 1% ./1per/tif/GB-006-def.1per.tif
convert -delete 1--1 ./original/tif/CN-053-def.tif -resize 40x400 ./40_400/png/CN-053-def.40_400.png
```
Note: the -delete 1--1 is due to the fact that some tiffs have 3 layers, one with the full profile, another with the profile without measurement tape and the third with a measurement tape.
To facilitate conversion we use the python script as command: `tiff2Images.py`
```
tiff2Images.py --makepath --inpath=/home/jesus/git/wsm.rest/imageProcessing/tmp/2019-10-15/original/tif --outpath=/home/jesus/git/wsm.rest/imageProcessing/tmp2
```
Note: the script uses `shell=True` (bad, bad, bad !!!)
Adding extra information
------------------------
Extra tags are generated but to include WOSIS, ISIS information. This is done with extra scripts. We have different information according to the possibilities of the image format since JPEG can have more tags than PNG and in turn tiff has only generic tags
**TIFF:**
```
e,g: gdalinfo NL-039-def.50per.tif
Metadata:
TIFFTAG_ARTIST=ISRIC - Theo Jacobs
TIFFTAG_COPYRIGHT=ISRIC - CC BY 4.0 http://creativecommons.org/licenses/by/4.0/
TIFFTAG_DOCUMENTNAME=ZM-010-def.50per.tif
TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
TIFFTAG_XRESOLUTION=96.000008
TIFFTAG_YRESOLUTION=96.000008
```
**JPEG:**
```
e.g: exif NL-039-def.50per.jpg
EXIF tags in 'FR-002-def.50per.jpg' ('Intel' byte order):
--------------------+----------------------------------------------------------
Tag |Value
--------------------+----------------------------------------------------------
Image Description |WOSIS:63625,ISIS:FR002
X-Resolution |72
Y-Resolution |72
Resolution Unit |Inch
User Comment |WOSIS:63625,ISIS:FR002
Exif Version |Exif Version 2.1
FlashPixVersion |FlashPix Version 1.0
Color Space |Internal error (unknown value 65535)
GPS Tag Version |2.0.0.0
North or South Latit|N
Latitude |48, 7, 59,999999
East or West Longitu|E
Longitude | 7, 3, 43,999999
Altitude Reference |Sea level7
Altitude |1007,700000
Geodetic Survey Data|WGS-84
--------------------+----------------------------------------------------------
```
**PNG:**
```
identify -verbose ZM-004-def.50per.png
Properties:
date:create: 2018-09-13T12:39:44+02:00
date:modify: 2018-09-12T13:49:56+02:00
exif:Artist: ISRIC - Theo Jacobs
exif:Copyright: ISRIC - CC BY 4.0 http://creativecommons.org/licenses/by/4.0/
exif:ExifOffset: 192
exif:GPSAltitude: 1771100000/1000000
exif:GPSAltitudeRef: 0
exif:GPSInfo: 240
exif:GPSLatitude: 8/1, 55/1, 1200/1000000
exif:GPSLatitudeRef: S
exif:GPSLongitude: 31/1, 16/1, 59998799/1000000
exif:GPSLongitudeRef: E
exif:GPSMapDatum: WGS-84
exif:GPSVersionID: 2, 0, 0, 0
exif:ImageDescription: WOSIS:63211,ISIS:ZM004
exif:UserComment: 0, 0, 0, 0, 0, 0, 0, 0, 87, 79, 83, 73, 83, 58, 54, 51, 50, 49, 49, 44, 73, 83, 73, 83, 58, 90, 77, 48, 48, 52
Profiles:
Profile-exif: 412 bytes
Profile-iptc: 40 bytes
Caption[2,120]: WOSIS:63211,ISIS:ZM004
```
To check the image information we can use the following commands:
```
# for png and also tif
identify -verbose AU-002-def.40_400.png
# for jpeg we have exif
exif KE-051a-def.80_800.jpg
```
Python scripts
==============
There are several python script that can be run according to their functionalities. They require `GExiv2` library, which can be a problem to install. The best strategy is using packages:
```
sudo apt install python3-gi python3-gi-cairo gir1.2-gtk-3.0
python3.6 -c "from gi.repository import GExiv2"
```
**Jorge, Let Op!!** your default python is 3.7 and therefore will not see GExiv2
Database information
--------------------
First step on processing is to get the profiles from WOSIS into a CSV:
```
select distinct p.country_id,p.geom,p.profile_id ,p.profile_code,cl.classification
from web_isis.vw_isis_profile as p
left join web_isis.vw_isis_wrb_classification_profile as cl ON cl.profile_id=p.profile_id
where (p.geom is NOT NULL and cl.classification<>'');
```
Notice that the Postgresql export makes CSV with `;` and we need to change it or save it on a proper format
Script profilesCodeGeomPicture.py
---------------------------------
This script converts the output CSV into a correct CSV with extra information. The script will crawl the files present and generate a CSV with filename and only with the images found. Therefore the script needs to be set with proper file names and outputs. For example the DB information like `PE021` needs to be transformed into `PE-021-def.80_800.tif`
TODO: Change this script into a python command using click package and rename it
Script addCopyright.py
---------------------
This script will add metadata content mainly *Artist* and *Copyright*
Script addCoordsComments.py
---------------------------
This script will add coordinates + comments, this script will use the CSV file with filenames to set the properties. It had some geocoding capabilities but is to old to be properly working
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment