pynitride.core.machine module

Utilities to manage parallelism via multiprocessing.

Importing this module (as is done automatically by importing pynitride, configures the environment in accordance with config.py (by default, this enables PyNitride to parallelize with as many processes as CPUs, and prevents numpy/ scipy or other common c-extensions which PyNitride uses from employing any internal parallelization.

Worker pools of both the process and thread variety are available by calling process_pool() and thread_pool() respectively. A FakePool class is provided to mimic either of these without actually creating more processes or threads. A no_parallel() context manager is provided to temporarily make the other functions use FakePools instead of real pools.

Functions such as glob_store() enable large objects to be stored by a reference so that they don’t get pickled when spawning new processes. This utility is further enhanced by the glob_store_attributes() wrapper which can automatically employ the glob_store() functions behind the scenes for specified large attributes so that the object can be safely pickled.

class pynitride.core.machine.Counter(print_every=10, print_message='Count: {}')

Bases: object

Provides a thread-safe counter.

Parameters:

  • print_every – every time this many increments has been met or passed, log a message

  • print_message – the message to log

increment(inc=1)

Increments the counter by inc.

class pynitride.core.machine.FakePool

Bases: object

Same API as multiprocessing.pool.Pool but actually applies serially, no processes/threads.

apply(func, args=(), kwds={})
apply_async(func, args=(), kwds={}, callback=None, error_callback=None)
close()
join()
map(func, iterable, chunksize=1)
starmap(func, iterable, chunksize=1)
pynitride.core.machine.glob_read(key)

Returns an object from the global storage.

Note: when multiprocessing, be aware that changes made in one process do not affect a previously spawned worker process. Use new=True with process_pool() in the parent process to incorporate recent updates.

Parameters:

key – returned from glob_store()

Returns:

the stored object

pynitride.core.machine.glob_remove(key)

Remove an object from the global storage.

Note: if called a child process other than the one which placed this object on the store, this method will not even attempt removal.

Note: when multiprocessing, be aware that changes made in one process do not affect a previously spawned worker process. Use new=True with process_pool() in the parent process to incorporate recent updates.

Parameters:

key – returned from glob_store()

Returns:

None

pynitride.core.machine.glob_store(obj)

Add obj to the global storage.

Note: when multiprocessing, be aware that changes made in one process do not affect a previously spawned worker process. Use new=True with process_pool() in the parent process to incorporate recent updates.

Parameters:

obj – anything.

Returns:

a key which can be used in glob_read() or glob_write() or glob_remove().

pynitride.core.machine.glob_store_attributes(*attrs)

Class decorator to automatically store certain attributes in the glob_store system.

For example:

@globstore_attributes(big_obj)
class MyClass:
    def __init__(self):
        self.big_obj = np.empty([1e6,1e6])

Now MyClass.big_obj can be get and set like any other parameter, but behind the scenes a property is in place such that the MyClass instance does not actually hold a reference to big_obj (just the key to retreive it from the glob_store system). Thus if an instance of MyClass is sent through a multiprocessing function and gets pickled, big_obj will not be shared through the pickling but instead through process inheritance.

Note: when subclassing a class which uses this wrapper, make sure that, if the subclass implements an __init__ or __del__ function, these implementations call the superclass __init__ or __del__ functions.

pynitride.core.machine.glob_update(key, obj)

Updates an object in the global storage.

Note: when multiprocessing, be aware that changes made in one process do not affect a previously spawned worker process. Use new=True with process_pool() in the parent process to incorporate recent updates.

Parameters:
Returns:

None

pynitride.core.machine.no_parallel()

Context manager to temporarily disable PyNitride-based parallelism.

pynitride.core.machine.parallel_enabled()

Returns whether parallelism is enabled in this context, see no_parallel()

pynitride.core.machine.process_pool(new=False)

Returns a pool of worker processes.

If parallelism is enabled in this context, this will be a multiprocessing.pool.Pool. Otherwise, it will be an object which superficially implements the same methods, but runs tasks serially.

Parameters:

new – if returning a real Pool, then refresh it first

Returns:

an object at least superficially resembling Pool

pynitride.core.machine.raiser(e)

Trivial functional form of the raise keyword

pynitride.core.machine.thread_pool()

Returns a pool of worker threads.

If parallelism is enabled in this context, this will be a Pool. Otherwise, it will be an object which superficially implements the same methods, but runs tasks serially.

Returns:

an object at least superficially resembling Pool