BlacklistUpdater fails to create blacklist.txt on new install #6

New Issue

2020-04-07T21:05:05+02:00

reinhard-mueller commented

2020-04-07 21:05:05 +02:00

(Migrated from github.com)

On a fresh install, where blacklist.txt is not yet present, BlacklistUpdater.process() fails because it tries to lock that non-existent file.

I see that setup.py should run BlacklistUpdater.process(), but it doesn't, at least not when installing with pip install. And if it would, it would obviously fail for the same reason anyway.

On a side note, I think that putting the updateable file inside the validate_email directory might lead to problems because on a system install, that directory might not be writeable for the process that actually runs the code.

On a fresh install, where blacklist.txt is not yet present, `BlacklistUpdater.process()` fails because it tries to lock that non-existent file. I see that setup.py should run `BlacklistUpdater.process()`, but it doesn't, at least not when installing with `pip install`. And if it would, it would obviously fail for the same reason anyway. On a side note, I think that putting the updateable file inside the validate_email directory might lead to problems because on a system install, that directory might not be writeable for the process that actually runs the code.

karolyi commented

2020-04-08 10:42:04 +02:00

(Migrated from github.com)

Hey,

thanks for reporting this issue. I will be testing this out soon along with the read-only filesystem problem and provide an update.

Probably I'll include a change where the blacklist.txt will get put into a /tmp like filesystem part (anything python suggests with tempfile).

Hey, thanks for reporting this issue. I will be testing this out soon along with the read-only filesystem problem and provide an update. Probably I'll include a change where the blacklist.txt will get put into a `/tmp` like filesystem part (anything python suggests with `tempfile`).

👍 1

reinhard-mueller commented

2020-04-08 17:53:09 +02:00

(Migrated from github.com)

I just did some further debugging and found out that the file lock does not fail because of the non-existent file, but rather because, in my test environment, the file would have been within an NFS mounted directory tree. If I move everything to a directory tree on the local hard disk, it works as expected.

I still think that putting it on /tmp is a good idea.

I just did some further debugging and found out that the file lock does not fail because of the non-existent file, but rather because, in my test environment, the file would have been within an NFS mounted directory tree. If I move everything to a directory tree on the local hard disk, it works as expected. I still think that putting it on `/tmp` is a good idea.

karolyi commented

2020-04-08 18:39:57 +02:00

(Migrated from github.com)

Are you saying that you can't create and then lock a file on NFS?

I think I'll change the lock to lock the updater.py itself, so it's an already existing file. The question is, can you lock an existing file via your NFS? I think both the creation and lock should work.

It's important to lock because it's the only way of avoiding multiple updates when more than one processes are fired up in the same time (e.g. UWSGI starts instances of code that use this module).

Are you saying that you can't create and then lock a file on NFS? I think I'll change the lock to lock the updater.py itself, so it's an already existing file. The question is, can you lock an existing file via your NFS? I think both the creation and lock should work. It's important to lock because it's the only way of avoiding multiple updates when more than one processes are fired up in the same time (e.g. UWSGI starts instances of code that use this module).

reinhard-mueller commented

2020-04-08 23:19:51 +02:00

(Migrated from github.com)

I found the root of the problem: see https://manpages.debian.org/buster/manpages-dev/flock.2.en.html section "NFS details":

It also means that in order to place an exclusive lock, the file must be opened for writing.

And indeed, if I change

with open(self._lock_file_path) as fd:

into

with open(self._lock_file_path, "a") as fd:

it works perfectly.

Still it might be more portable to different situations and operating systems if you use, for example, https://pypi.org/project/filelock/.

I found the root of the problem: see https://manpages.debian.org/buster/manpages-dev/flock.2.en.html section "NFS details": > It also means that in order to place an exclusive lock, the file must be opened for writing. And indeed, if I change ```python with open(self._lock_file_path) as fd: ``` into ```python with open(self._lock_file_path, "a") as fd: ``` it works perfectly. Still it might be more portable to different situations and operating systems if you use, for example, https://pypi.org/project/filelock/.

Sign in to join this conversation.