mmfutils.containers

Provides convenience containers that support pickling and archiving.

Archiving is supported through the interface defined by the persist package (though use of that package is optional and it is not a dependency).

class mmfutils.containers.Container(*argv, **kw)[source]

Bases: mmfutils.containers.Object, collections.abc.Sized, collections.abc.Iterable, collections.abc.Container

Simple container object.

Attributes can be specified in the constructor. These will form the representation of the object as well as picking. Additional attributes can be assigned, but will not be pickled.

Examples

>>> c = Container(b='Hi', a=1)
>>> c                       # Note: items sorted for consistent repr
Container(a=1, b='Hi')
>>> c.a
1
>>> c.a = 2
>>> c.a
2
>>> tuple(c)                # Order is lexicographic
(2, 'Hi')
>>> c.x = 6                 # Will not be pickled: only for temp usage
>>> c.x
6
>>> 'a' in c
True
>>> 'x' in c
False
>>> import pickle
>>> c1 = pickle.loads(pickle.dumps(c))
>>> c1
Container(a=2, b='Hi')
>>> c1.x
Traceback (most recent call last):
...
AttributeError: 'Container' object has no attribute 'x'
class mmfutils.containers.ContainerDict(*argv, **kw)[source]

Bases: mmfutils.containers.Container, collections.abc.MutableMapping

Simple container object that behaves like a dict.

Attributes can be specified in the constructor. These will form the representation of the object as well as picking. Additional attributes can be assigned, but will not be pickled.

Examples

>>> from collections import OrderedDict
>>> c = ContainerDict(b='Hi', a=1)
>>> c                       # Note: items sorted for consistent repr
ContainerDict(a=1, b='Hi')
>>> c['a']
1
>>> c['a'] = 2
>>> c.a
2
>>> OrderedDict(c)
OrderedDict([('a', 2), ('b', 'Hi')])
class mmfutils.containers.ContainerList(*argv, **kw)[source]

Bases: mmfutils.containers.Container, collections.abc.Sequence

Simple container object that behaves like a list.

Examples

>>> c = ContainerList(b='Hi', a=1)
>>> c                       # Note: items sorted for consistent repr
ContainerList(a=1, b='Hi')
>>> c[0]
1
>>> c[0] = 2
>>> c.a
2
>>> tuple(c)                # Order is lexicographic
(2, 'Hi')
class mmfutils.containers.Object(**kw)[source]

Bases: mmfutils.containers.ObjectMixin, mmfutils.containers.ObjectBase

Extension of Object with pickling support.

Pickling will save only those variables defined in picklable_attributes which is usually defined when the base __init__ is finished. The init() method will be called upon unpickling, thereby allowing unpicklable objects to be used (in particular function instances).

Note

Do not use any of the following variables:

  • _empty_state:

    Reserved for objects without any state

  • _independent_attributes:

  • _dependent_attributes:

  • _strict:

    If True, then only picklable attributes will be settable through __setattr__().

  • _check:

    If True, check that objects are actually picklable when they are set.

  • _reserved_attributes:

    List of special attributes that should be excluded from processing.

To allow for some variables to be set without invalidating the object we also check the set of names _independent_attributes.

Examples

>>> class A(Object):
...     def __init__(self, x=0):
...         self.x = x
...         super().__init__()
...     def init(self):
...         self.x1 = self.x + 1   # A dependent variable
...         super().init()
...     def check(self):
...         if not self.initialized:
...             raise AssertionError("Please call init()!")
...         return self.x1 == self.x + 1
>>> a = A(x=0)
>>> a.check()
True
>>> a.x = 2.0
>>> a.check()
Traceback (most recent call last):
...
AssertionError: Please call init()!
>>> a.init()
>>> a.check()
True
class mmfutils.containers.ObjectBase(**kw)[source]

Bases: object

General base class with a few convenience methods.

Summary:

  • __init__() sets parameters and calls init()

  • init() calculates all other parameters.

Motivation:

The motivation is objects intended to be used in computationally demanding settings. The idea is that the init() method will be called before starting a computation, ensuring that the object is up-to-date, and performing any expensive calculations. Then the object can be used in a computationally demanding setting.

I have been using this approach for some time and am generally happy with how it works. Some care is needed nesting calls to init() in derived classes, but I have found these cases easy to deal with. Other approaches such as using properties can carry a performance hit. Writing setters can work well, but demands a lot from the developer and become very complicated when properties depend on each other.

Details:

  • The constructor __init__() should only be used to set variables in self. The reason is that the code here uses the variables set in the constructor to determine which attributes need to be pickled. Initialization of computed attributes should instead be done in the init() method .

  • The constructor __init__() takes kwargs and will set these. This allows using super().__init__(). See e.g.:

  • The constructor will store all assigned variables (in __dict__()) to a list picklable_attributes which can then be used by the Object to provide pickling services. Here we simply use this to set an initialized flag to note the user that the object might be invalid and need init() called again.

  • The init() method should make sure that the object ends in a consistent state so that further computations (without users setting attributes) can be computed efficiently. If the user sets attributes, init() should be called again.

Note

Do not use any of the following variables:

  • picklable_attributes:

    Reserved for the list of attributes that will be pickled. If this has been stored in self.__dict__ then the constructor chain has finished processing.

  • initialized:

    Used to flag if attributes have been changed but without init() being called.

By default setting any attribute in picklable_attributes will set the initialized flag to False. This will be set to True when init() is called. Objects can then include an assert self.initialized in the appropriate places.

Note

This redefines __setattr__ to provide the behaviour.

Examples

>>> class A(ObjectBase):
...     def __init__(self, x=0):
...         super().__init__(x=x)
...     def init(self):
...         self.x1 = self.x + 1   # A dependent variable
...         super().init()
...     def check(self):
...         if not self.initialized:
...             raise AssertionError("Please call init()!")
...         return self.x1 == self.x + 1
>>> a = A(x=0)
>>> a.check()
True
>>> a.x = 2.0
>>> a.check()
Traceback (most recent call last):
...
AssertionError: Please call init()!
>>> a.init()
>>> a.check()
True
__setattr__(key, value)[source]

Sets the initialized flag to False if any picklable attribute is changed.

get_persistent_rep(env)[source]

Return (rep, args, imports).

Define a persistent representation rep of the instance self where the instance can be reconstructed from the string rep evaluated in the context of dict args with the specified imports = list of (module, iname, uiname) where one has either import module as uiname, from module import iname or from module import iname as uiname.

This satisfies the IArchivable interface for the persist package.

init()[source]

Initialize Object.

initialized = False
picklable_attributes = ()