Crashes when meets it's own temp file

[varnav]

I often get errors like this:

✅  [OPTIMIZED] .\22may_shaslyk\22may_shaslyk 051.jpg
     JPG/RGB: 622.0 KB  ->  JPG/RGB: 345.3 KB 🔻 44.5%concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "c:\users\someone\appdata\local\programs\python\python37\lib\shutil.py", line 566, in move
    os.rename(src, real_dst)
FileExistsError: [WinError 183] Невозможно создать файл, так как он уже существует: '.\\22may_shaslyk/~temp~22may_shaslyk 052.jpg' -> '.\\22may_shaslyk\\22may_shaslyk 052.jpg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\someone\appdata\local\programs\python\python37\lib\concurrent\futures\process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "c:\users\someone\appdata\local\programs\python\python37\lib\concurrent\futures\process.py", line 198, in _process_chunk
    return [fn(*args) for args in chunk]
  File "c:\users\someone\appdata\local\programs\python\python37\lib\concurrent\futures\process.py", line 198, in <listcomp>
    return [fn(*args) for args in chunk]
  File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\optimize_images\__main__.py", line 67, in do_optimization
    return optimize_jpg(t)
  File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\optimize_images\img_optimize_jpg.py", line 104, in optimize_jpg
    shutil.move(temp_file_path, os.path.expanduser(t.src_path))
  File "c:\users\someone\appdata\local\programs\python\python37\lib\shutil.py", line 580, in move
    copy_function(src, real_dst)
  File "c:\users\someone\appdata\local\programs\python\python37\lib\shutil.py", line 266, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "c:\users\someone\appdata\local\programs\python\python37\lib\shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
OSError: [Errno 22] Invalid argument: '.\\22may_shaslyk\\22may_shaslyk 052.jpg'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\users\someone\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\someone\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\someone\AppData\Local\Programs\Python\Python37\Scripts\optimize-images.exe\__main__.py", line 9, in <module>
  File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\optimize_images\__main__.py", line 96, in main
    for r in executor.map(do_optimization, tasks):
  File "c:\users\someone\appdata\local\programs\python\python37\lib\concurrent\futures\process.py", line 483, in _chain_from_iterable_of_lists
    for element in iterable:
  File "c:\users\someone\appdata\local\programs\python\python37\lib\concurrent\futures\_base.py", line 598, in result_iterator
    yield fs.pop().result()
  File "c:\users\someone\appdata\local\programs\python\python37\lib\concurrent\futures\_base.py", line 435, in result
    return self.__get_result()
  File "c:\users\someone\appdata\local\programs\python\python37\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
OSError: [Errno 22] Invalid argument: '.\\22may_shaslyk\\22may_shaslyk 052.jpg'

Looks like it's a race condition. It's less likely to happen if system is under load. I'm running under Windows 10.

[victordomingos]

Hmm. I believe I have also seen that kind of error, but at the time I couldn’t determine what was causing it, probably because it was a rare occurrence in my case. I need to take another look at it, but you’re probably right concerning the possibility of this being a race condition.

Currently the temporary files are being created in the same folder that has the original file, with a prefix in the name. Maybe it needs to be set to ignore any image files whose name starts with that prefix. Or maybe the temporary file can be replaced with something in memory without disk I/O, which should be much faster and would probably solve this issue.

[varnav]

Looks like it's more likely to occur on fast hardware - I have 12 core CPU and NVMe SSD. I noticed that if system is under load - this is less likely to occur. Of course it would be the best not to create temp file at all. Maybe this will help: https://docs.python.org/3/library/tempfile.html#tempfile.SpooledTemporaryFile

[victordomingos]

Thank you very much for your feedback. I submitted a quick fix, just checking the presence of the temporary files prefix (~temp~) in the filename. It should be enough to avoid the reported issue. I will keep in mind your SpooledTemporaryFile suggestion, though.

Please let me know in case you still get any issue with this version.

[victordomingos]

Just a quick notice to let you know that, on the latest commit (not yet available in PyPI), I have implemented in-memory buffers using BytesIO, instead of in-disk temporary files. Still under testing, as the refactoring could have introduced some regression bugs.

I am still curious about any potential advantages of SpooledTemporaryFile over BytesIO, though.