Michael Medin

Random thoughts on development, SOA and monitoring…

Thumbnail 1000:s of photos on a Synology NAS (in hours not months)

synology_ds212I moved all my photos over to my Synology NAS (which I really enjoy) just to discover that it would take months to index all photos. So I started to look for options and while most people managed to increase the speed a bit by reducing the quality of the thumbnails I figured why not utilize some of the CPU power I have at home?

So I ended up using my Linux box to do the thumbnailing for me instead which reduced the time to thumbnail 30.000 photos down to hours.

The process

The process is simple and straight forward and requires NFS as well as ImageMagic on the Linux box I also disabled the indexer on the Synology DS (using the pause feature) but I don’t think this is strictly required.

The first step is to mount the photo share:

mkdir /mnt/photo
sudo mount DS_IP:/volume1/photo /mnt/photo/

Second step is to create a script to generate thumbnails.

This will for all *.jog create the required thumbnails for you if you have other files you might need to tweak this.

#!/bin/bash
pushd "$1"
shopt -s nocaseglob
if [ ! -d @eaDir ] ; then mkdir @eaDir ; fi
for f in *.jpg ; do
if [ "$f" == "*.jpg" ] ; then break ; fi
echo "$1 - $f..."
if [ ! -d @eaDir/$f ] ; then mkdir @eaDir/$f ; fi
if [ ! -f @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ] ; then convert $f -resize 1280x1280\> -quality 90 -unsharp 0.5x0.5+1.25+0.0 @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ; fi
if [ ! -f @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ] ; then convert @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg -resize 800x800\> -quality 90 -unsharp 0.5x0.5+1.25+0.0 @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ; fi
if [ ! -f @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ] ; then convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 320x320\> -quality 90 -unsharp 0.5x0.5+1.25+0.0 @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ; fi
if [ ! -f @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ] ; then convert @eaDir/$f/SYNOPHOTO:THUMB_M.jpg -resize 120x120\> -quality 90 -unsharp 0.5x0.5+1.25+0.0 @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ; fi
if [ ! -f @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ] ; then convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 640x640\> -quality 90 -unsharp 0.5x0.5+1.25+0.0 @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ; fi
done
popd

Last step is to run the script in the folder you want to thumbnail (or the root if you want to thumbnail it all):

Replace ~/tn.sh with the location of the script you created before.

find . -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec ~/tn.sh {} \;

A big warning

Now this may void you warranty break your NAS and generally end the world as you know it. I tried it on my NAS and it works but there are no guarantees. But I hope that the people at Synology will listen to this and create a sanctioned way to remotely create thumbnails as it is much better (especially as upgrading photo station requires re-indexing) than the rather cheep re-upload photos using a tool solution they have today.

Some theory

So what does this do? And how do the thumbnails work?

Well that is pretty straight forward.

If you create a folder called folder 1 with two photos “photo 1.jpg” and “photo 2.jpg” the indexer will create the following folder structure:

image

In other words a folder called @eaDir with a subfolder for each photo. The subfolder representing a photo in turn contain the various thumbnails.

The thumbnail process is configured in the following file /usr/syno/etc/thumb.conf or /usr/syno/etc/thumb_high.conf depending on which setting you have. I use the normal one (low resolution thumbnails) which is defined like so:

size=120, quality=90, unsharp=0.5x0.5+1.25+0.0, filename=SYNOPHOTO:THUMB_S.jpg
size=320, quality=90, unsharp=0.5x0.5+1.25+0.0, filename=SYNOPHOTO:THUMB_M.jpg
size=640, quality=80, unsharp=0.5x0.5+1.25+0.0, filename=SYNOPHOTO:THUMB_B.jpg
size=800, quality=90, unsharp=0.5x0.5+1.25+0.0, filename=SYNOPHOTO:THUMB_L.jpg
size=1280,quality=90, unsharp=0.5x0.5+1.25+0.0, filename=SYNOPHOTO:THUMB_XL.jpg

Which is essentially just command line arguments for ImageMagicks convert command (which is what I call in my script above).

Samba anyone?

A quick note is that the reason this doesn’t work via samba (smb) I think is the colon character. A colon in a smb filename (at least in windows) is a stream A pretty cool but very neglected feature of NTFS. A stream is a file in a file meaning you can put more than one stream of data in each file. This was a cool way to “joke” with co-workers back in the Windows NT4 days as Explorer did not see streams. If you created a file somewhere with 0 bytes and then put megabytes of data in a stream the file would still be listed as 0 bytes yet take up all the space on the hard disk Ler

Category: Personal
  • Daniel Wittenberg says:

    What software are you using that utilizes the thumbnails? We currently use iPhoto for all our picts and small videos, and just back them up to a Linux RAID + fireproof USB drive, but I was thinking something that we could manage them directly on the Linux box would be handy too. I had Gallery setup at one point, but not used it since rebuilding my server about a year ago. We don’t have as many as you, only about 22k but seems to be growing at a staggering rate each year > 5k/year now.

    2012-04-23 at 00:04
    • Michael Medin says:

      Well, Synology has a web gallery (which I don’t really use) but I use their android app a bit which means I have all my photos ”in my android phone” (read in the cloud easily accessible from the android phone).
      But they main feature for me is UPnP which means all photos are available on the xbox, boxee box and what not so I can display them on the TV. And the main benefit with having them resized a bit is that it is blazing fast to view them on the TV…

      2012-04-23 at 06:58
  • Tomas says:

    thanks!!
    a great script – it works fine for 50.000 pictures
    I used a ubuntu live cd to boot my windows machine…

    I have a quad core cpu but I’m not succesfull with trying convert multiple pictures at once in background (convert .. &; wait $!)

    2012-05-12 at 03:56
  • Tommy says:

    Hi, Great post….

    I dont have a linux box so I am working on a windows DOS script. I am having problem with the parameter:

    filename=SYNOPHOTO:THUMB_B.jpg

    I believe windows doesnt allow the : (colon) in the filename. Any in-site or work around will be appreciated.

    thanks.

    2012-05-17 at 09:51
    • Michael Medin says:

      The : is indeed the problem which is why I did this via NFS from Linux. I am planing to look into this this weekend to see if I can do a simple Java version which would work on all platforms.

      2012-05-18 at 06:54
  • Arnaud says:

    Hi Michael, thank you very much for this post. I am the proud owner of a DS212j since 4 days, and while setting up the box quickly stumbled over the poor performance fo the thumbnail created. Found many other people complaining, and some suggestions, but this does the trick for me. I have uploaded 50 GB of photos. Glad I found your article to make this less painful.

    2012-06-17 at 22:37
  • Brian says:

    I don’t think the ”:” problem is solvable at all using Windows. You simply cannot create a file containing a ”:” in it. My solution was to use a script similar to above (running with cygwin), but instead of ”:”, I use an ”_”. I then run a very small, simple script on the synology itself, that renames the files to have the ”:”. I ssh in and run the script after my cygwin process completes. Here’s the script:

    if [ "$1" != "FROG" ]; then
    find . -type d -name ”*.JPG” -exec ./fix FROG {} \;
    exit
    fi

    echo processing $2 directory
    mv ”$2″/SYNOPHOTO_THUMB_B.jpg ”$2″/SYNOPHOTO:THUMB_B.jpg
    mv ”$2″/SYNOPHOTO_THUMB_M.jpg ”$2″/SYNOPHOTO:THUMB_M.jpg
    mv ”$2″/SYNOPHOTO_THUMB_L.jpg ”$2″/SYNOPHOTO:THUMB_L.jpg
    mv ”$2″/SYNOPHOTO_THUMB_S.jpg ”$2″/SYNOPHOTO:THUMB_S.jpg
    mv ”$2″/SYNOPHOTO_THUMB_XL.jpg ”$2″/SYNOPHOTO:THUMB_XL.jpg

    2012-06-20 at 10:06
  • fippiko says:

    Works fine with my DS110j.
    Thank you very much!

    2012-08-08 at 16:41
  • uros says:

    Thanks a lot for this. I enabled a photo station package about 5 days ago and it was converting and converting for 5 days and nothing was done, so I have to find your great solution…

    Thanks..

    2012-10-01 at 20:21
  • Mikko says:

    Here is something a little bit more efficient in c++ you can use to unconditionally create all thumbnails for a file, it only sharpens once, you can change this if you like… It only reads the file once which makes a difference… esp in a network. You also don’t get the loss from recompressing jpeg many times as done in the above script.

    https://gist.github.com/3839394

    2012-10-05 at 13:55
  • Mikko says:

    But honesty, moving all photos to another location outside photos folder but available to your other more powerful computer… then using Synology Assistant to upload everything again might be just as good/fast…

    2012-10-05 at 14:39
    • Michael Medin says:

      Not sure you can create the ”hidden folders” when you move them back though?
      Or did you mean to use the assistant to create the thumbnails?

      2012-10-11 at 11:25
  • quietnan says:

    You might have to run ”synoindex -U” on the updated folders to update the database and put your changes into effect.

    2012-10-24 at 23:14
  • Paul says:

    Great script! Cut my indexing time for a test folder of 38 files from 30 minutes using Synology’s reindexing process to 35 seconds. However, thumbnail files are about 10% larger than with Synology’s process.

    I did note one glitch. Sometimes, my wife will edit a photo and then save it with an embedded space, e.g. ”april visit.jpg”. In that case, the thumbnail folder and thumbnail files are not created. Here are the errors:

    /home/paul/make_thumbs.sh: line 8: [: @eaDir/april: binary operator expected
    /home/paul/make_thumbs.sh: line 9: [: @eaDir/april: binary operator expected
    /home/paul/make_thumbs.sh: line 10: [: @eaDir/april: binary operator expected
    /home/paul/make_thumbs.sh: line 11: [: @eaDir/april: binary operator expected
    /home/paul/make_thumbs.sh: line 12: [: @eaDir/april: binary operator expected
    /home/paul/make_thumbs.sh: line 13: [: @eaDir/april: binary operator expected

    Any thoughts on how to address this? Could there be other ”illegal” characters that might get into a file name that would cause a similar problem?

    2012-11-01 at 23:37
  • Mark says:

    These scripts + Mikko’s c++ program saved me a few days of indexing – thanks!

    Mikko, it may be just my system but I had to #include to get the code to compile.

    After that, it was just a matter of replacing the calls to convert in the original script. I also added a check that each image hand’t already been processed, like so:

    dirName=@eaDir/$1
    if [ ! -d "$dirName" ] ; then mkdir $dirName ; fi
    if [[ -z `ls -A "$dirName"` ]]; then
    /path/to/synthumbs $1
    else
    echo $1 already processed
    fi

    I also got it going in parallel, by replacing the for loop with a call to gnu parallel (I had to split out the inside of the loop to a separate script so it could be called from gnu parallel).

    2012-11-02 at 00:55
  • Mark says:

    I was trying to say I had to #include cstring.

    2012-11-02 at 00:56
  • Paul says:

    Comment to my post above regarding errors with files containing spaces . . .

    I replaced $f with ”$f” in lines 8 through 13 in the original script. That change resulted in no errors and the successful creation of the thumbnail folders and files.

    Hope this may be of help to someone else.

    2012-11-03 at 22:02
    • Michael Medin says:

      Thanks for a good point and suggestion!

      I was a bit lazy in regards to funny file names.
      using ”…$f…” will work for many (but not all) cases though for instance I would guess a file with ” in it would probably not work.

      But that is the main problem with shell scripts really (and why I was planning to make a java version, but I ever got around to it :)

      // Michael Medin

      2012-11-06 at 09:58
  • Nick says:

    I used all this input and created an update on this with details I really needed to know for my usecase.
    Also the C++ script was improved and made more fault-tolerant. I pasted the source on my blog.

    http://nickveenhof.be/blog/speed-thumbnail-generation-synology-ds212j

    Thanks for opening up all this information!

    2012-11-25 at 21:22
  • Karo says:

    A good choice to run your script. I have 80k photos ;)
    After a wait for 2 days 99% CPU all the time I decided google it what to do.

    TY man!

    2012-12-25 at 11:27
  • Tom says:

    I tried the script for my DS212j but after thumb’ing all pictures the ‘convert-thumb’ process still is eating up my CPU power of the NAS. First I was confused about the little information you gave us here… so I will show you how I did it:

    *Maybe you find the problem why the ‘convert-thumb’ process is still running?*

    1) NAS > activate NFS
    2*) NAS > Shared folders > select ‘photos’ > Privileges > create a new NFS share for read/write access using the IP address of my laptop (e.g. 192.168.0.100, the asterisk 192.168.0.* seems not to work) and mapping root squash to the admin user
    3) NAS > stop Media server (package center) to stop indexing
    4) download Ubuntu Linux CD and boot it in my laptop (try, not install)
    5*) in Terminal, set a root password or no `su` or `sudo` will work: `sudo passwd root` (I set ‘root’ as the root’s password), then I just did `su` to work with root privileges
    6*) install the packages for NFS and ImageMagick: `apt-get install nfs-common imagemagick` (or use Ubuntu’s software center to find them); the package `portmap` is also needed but also pre-installed on Ubuntu
    7) mount the NFS drive: `mkdir /mnt/photo` and `mount 192.168.0.100:/volume1/photo /mnt/photo/`
    8) create a new empty file ‘tn.sh’ in my personal folder (/home/ubuntu)
    9) copy your code into the file and save it
    10) right-click the file > properties > run as program (I guess doing that sets the execute-rights)
    11) run the script: `find /mnt/photo -type d -name @eaDir-prune -o ! -name @eaDir-type d -exec /home/ubuntu/tn.sh {} \;`

    ^^ I directly used ‘/mnt/photo’ instead of ‘.’ as starting point and used the absolute path ‘/home/ubuntu/tn.sh’ for the script’s location. Is that OK so far?

    *) I didn’t know that I have to do that

    Then I could see the script running throught the NAS file structure and creating thumbnails for each JPG image so it seemed to work right.

    After that (about 2 hours for 40.5 GB of photos = 12.547 files) I re-started the Media server and the ‘convert-thumb’ process came up immediately by consuming ~90% of my CPU power :-(

    A big disappointment so far… I’m using the latest DSM software 4.1-2668.

    First I thought the ‘convert-thumb’ process came up again for all the none-JPG-images but there are just a few of them. I looked over SSH and the `ps w | grep convert-thumb` command to watch the process but it doesn’t show me the full path of the processed image.

    Any hints for me how to solve this? Thanks so much for your idea and help!

    2012-12-30 at 15:11
  • droop says:

    Hi,

    I was looking exactly for this solution ! You saved me days and weeks of processing with this !
    Now I can finally use the nice photo station android application !

    Just a small change to the convert command to be fully compatible with Photostation , you must add the ”-auto-orient” option to have the right orientation in Photostation app and website.

    Thanks again !

    droop

    2013-01-17 at 00:56
  • Droop says:

    Hi,

    I had the same problem with several weeks of processing for my photos, and I wondered how to use PC CPU to do that. That is the solution I expected !

    Just one more point for people who use the photo station application (really nice client application for mobile and tablets), you must add the ”-auto-orient” parameter in the convert command line options, otherwise portrait photos will appear horizontally.

    Thanks again for this script !

    2013-01-17 at 12:08
  • Martin says:

    i love it

    2013-02-05 at 22:59
  • AJ says:

    Hi,
    First of all thanks to you all for such a wonderful program. I was finally able to execute Nick’s modified c++ program. I have few issues.
    1. Some folders are missing from photo station. For example I have 2012 folder and that has Europe11 under that and I can not see Europe11 on the photostation or that iPhone app. But I checked and it did create the thumbnails for that folder. What should I do to resolve this issue?
    2. I tried .NEF files and those do not work with this. How should I make it work?
    3. I see some thumbnails are dis oriented and I read a post say to add -auto-orient. But where should I add that to c++ program?
    4. Also should I delete all @eaDir folders first by using recursive command and then use c++ program?
    find . -name ”@eaDir” -type d -print0 | xargs -0 rm -rf
    Thanks for help.

    2013-03-15 at 15:00
  • Antonio says:

    Thank you very much and thanks to Nick too.
    Finally I have found a perfect working solution for this problem :)

    2013-03-15 at 17:08
  • Louis Somers says:

    To work around the samba problem in windows I checked how the official Photo Station Uploader was doing it. It’s calling [sd-IP]/photo/include/asst_file_upload.php with PUT, and it looks like it’s uploading all thumbnails and the original photo in one go.

    I’m not sure how to find the PHP script on my DS in order to reverse engineer it any farther (I’m more at home in Windows than in Linux). I guess it wouldn’t be too hard to utilize the same php script for uploading the thumbnails?

    2013-04-27 at 14:14

Your email address will not be published. Required fields are marked *

*