Time Dependent Nodes¶

Nodes are marked as requiring re-calculation whenever any of their dependencies are modified. They are later lazily evaluated as required.

There’s a builtin node now() that behaves in a more specialized way and allows node valuations to evolve over time.

When the now() node is advanced, which can be done manually via MDFContext.set_date(), all the nodes dependent on time are marked as requiring re-calculation but additionally they are marked that the reason they require re-calculation is because time has moved forwards.

evalnode() nodes can be generators instead of regular functions (generators yield values rather than return a single value). When a generator is used mdf will advance the generator of the node each time the now() node is advanced. This allows state to be maintained between valuations:

from mdf import MDFContext, evalnode, now
from datetime import datetime, timedelta

@evalnode
def time_dependent_node():
    """
    a simple node whose value is dependent on 'now'
    """
    # returns 0, 1, 2, ... for the current weekday
    return now().weekday()

@evalnode
def incrementally_updated_node():
    """
    the value of this node is the sum of another node
    """
    # the first value will simply be time_dependent_node
    todays_value = time_dependent_node()
    yield todays_value

    # when the date is advanced this generator is continued
    # until the next yield

    while True:
        # yield today's value + the value evaluated previously
        prev_value = todays_value
        todays_value = time_dependent_node()
        yield todays_value + prev_value

# create the context with an initial date
date = datetime(2011, 9, 2)
ctx = MDFContext(date)

# get the value of incrementally_updated_node
x = ctx[incrementally_updated_node]
# x is now 4 (Friday)

# advance the date one day
date += timedelta(days=1)

# set the date on the context to be invoked (this causes the
# incrementally_updated_node generator to be advanced)
ctx.set_date(date)

# get the value of incrementally_updated_node
x = ctx[incrementally_updated_node]
# x is now 9 : 4 (Friday) + 5 (Saturday) = 9

This is a simple example, but the same methods can be used to build more complex nodes that perform incrementally calculated time-dependent nodes.

If time is ever moved backwards by calling MDFContext.set_date() then the current state of the time dependent nodes is discarded and the initial state will be re-evaluated by restarting the generators.

from mdf import evalnode

def my_node_filter():
    # the filtered evalnode will only be advanced on business days
    if my_is_valid_business_day_function():
        return True
    return False

@evalnode(filter=my_node_filter)
def my_node():
    yield some_initial_value
    while True:
        do_some_update_calculation(...)
        yield updated_value

Queue Nodes¶

Queue nodes are a specialized time-dependent node. The value of the node is a double ended queue (see collections.deque) of values. A double ended queue is used as it supports efficient appending and popping to both sides of the queue. Queues can also be used to construct numpy arrays and regular python lists.

The node function is called each time the node now() is advanced and the result is appended to the queue. The value of the node is the queue itself, which should be regarded as immutable.

Below is an example that uses a queue to get a delayed value:

from mdf import evalnode, queuenode

@queuenode
def some_value_queue():
    # do some calcuations
    return result

@evalnode
def delayed_value():
    values = some_value_queue() # type is collections.deque
    if len(values) < 5:
        return np.nan

    # return the value calculated 4 timesteps ago
    # (the item at -1 is the value for now)
    return values[-5]

Queue nodes can be bounded so they don’t grow indefinetely. This is done by setting the size of the queue. Once the queue reaches that size older items will be popped off the queue. The size can be specified as either an integer value or as a callable object (e.g. function or node) which can be useful if the size if a function of another node. Once the queue is created the size is fixed for that context:

# keep at most 5 values
@queuenode(size=5)
def some_value_queue():
    # do some calcuations
    return result

#
# or calculate the size as a function (or node)
#

def get_queue_size():
    return 5

@queuenode(size=get_queue_size)
def some_other_value_queue():
    # do some calcuations
    return result

Because queue nodes are a specialization of the eval node, they may also be filtered in the same way. If a filter is applied only when the filter returns True will values be calculated and appended to the queue.

Other Node Types¶

While eval nodes can be used to calculate any type of value, commonly used valuation types can be packaged as other node types for convenience. Currently the list of these specialized nodes is quite small, but as more use-cases are presented it’s reasonable to expect this list to grow.

Because these nodes are all specializations of the eval node, they may also be filtered in the same way. If a filter is applied only when the filter returns True will values be calculated or updated.

from mdf import evalnode, delaynode

@delaynode(periods=10)
def a_delayed_value():
    return some_value

@evalnode
def some_other_value():
    x = a_delayed_value() # this is the valued returned by a_delayed_value as it
                          # was 10 timesteps ago

@delaynode(periods=1, initial_value=0)
def delayed_a():
    return a()

@evalnode
def a():
    return 1 + delayed_a()

@delaynode(periods=1, initial_value=0, lazy=True)
def delayed_a():
    return a()

from mdf import evalnode, nansumnode

@nansumnode
def some_value():
    return some_value

@evalnode
def sum_of_some_value():
    value_sum = some_value() # this is the sum of 'some_value' for all time steps so far

from mdf import evalnode, cumprodnode

@cumprodnode
def some_value():
    return some_value

@evalnode
def sum_of_some_value():
    value_prod = some_value()  # this is the cumulative product of 'some_value'
                                                       # for all time steps so far

A_plus_B = A.apply(operator.add, args=(B,))

A_plus_B_node = A.applynode(operator.add, args=(B,))

smoothed_A_plus_B = A_plus_B_node.cumprod(...)

Method Syntax For Node Types¶

Creating a new node for simple operations on an existing node can make code look bloated and difficult to follow.

For this reason every node type is also exposed as methods on all other nodes. This is a syntactic helper and the end result is exactly the same as if a new node using the node type decorator was used.

This is best illustrated by example:

from mdf import evalnode, cumprod
from random import random

@evalnode
def random_value():
    while True:
        yield random()

If we wanted to compute the cumulative product of this random value you could do it by creating a new node using the cumprod() decorator:

@cumprodnode(half_life=10)
def cumulative_product_of_random_value():
    return random_value()

But if there are many nodes this can become a bit awkward. Using the method syntax the same thing can be achieved as follows:

@evalnode
def some_other_node():
    ewam_of_random_value = random_value.cumprod(half_life=10)

    # do some more calculation
    return result

When the cumprod method on the random_value node is called an internal node is created for that cumulative product calculation. Each subsequent time it’s called that internal node is re-used and so the effect is exactly the same as if the cumprod node was created explicitly.

All of the standard node types have corresponding methods, and custom node types can optionally expose themselves as methods.

In addition, there is also a method that returns the internal implicitly created node. This allows for chaining, e.g.:

@evalnode
def some_other_node():
    ewam_of_random_value_node = random_value.cumprodnode(half_life=10)
    delayed_cumprod = ewam_of_random_value_node.delay(periods=10, initial_value=0)

    # do some more calculation
    return result

Or more simply:

@evalnode
def some_other_node():
    delayed_cumprod = random_value.cumprodnode(half_life=10).delay(periods=10, initial_value=0)

    # do some more calculation
    return result

Setting Values¶

Values are set for nodes in a context using the MDFContext.set_value() method or more commonly by indexing into the context with the node:

A = varnode()
ctx = MDFContext()

# set the value of A in ctx
ctx[A] = 100

# get the value A in ctx
a = ctx[A] # a == 100

Any node type can have a value set to it. Once a value is set for a node in a context it is fixed until it’s changed again. Even if a node has dependencies (e.g. an evalnode()) and its dependencies are changed the node won’t be re-calculated if a value has explicitly been set.

Once a context has been created and any initial values have been set any available nodes can be evaluated. Node values can also be set in the context and when any dependent nodes are re-evaluated they will reflect those changes

from mdf import MDFContext, evalnode, varnode, now
from datetime import datetime, timedelta

n_days_offset = varnode("n_days_offset")

@evalnode
def n_days_from_today():
    offset = n_days_offset()
    return now() + timedelta(days=offset)

# create the context and set the date
ctx = MDFContext()
ctx.set_date(datetime(2011, 8, 9))

# set the value of n_days_offset
ctx[n_days_offset] = 1

print ctx[n_days_from_today] # datetime.datetime(2011, 8, 10, 0, 0)

# update n_days_offset
ctx[n_days_offset] = 100

print ctx[n_days_from_today] # datetime.datetime(2011, 11, 17, 0, 0)

Overriding Nodes¶

As well as being able to set a node’s value in a context it is also possible to override a node itself in a context. For example, suppose you had the following graph

It’s possible to override node B with node C using the MDFContext.set_override() method, or by setting the value on the context for one node with another node:

from mdf import MDFContext, evalnode, varnode

B = varnode(default=10)
D = varnode(default=20)

@evalnode
def A():
   return B() * 5

@evalnode
def C():
   return D() * 10

ctx = MDFContext()

a = ctx[A] # a == 50

# override B with C
ctx[B] = C

a = ctx[A] # a == (B overriden with C) * 5
           #   == (D * 10) * 5 == 1000

# A now depends on C which depends on D
# so changing D changes A
ctx[D] = 2

a = ctx[A] # a == 100

The resulting graph (where := denotes overriden with) looks like this:

Any type of node can be overriden by any other type of node. This means that you can override entire sub-graphs or add new sub-graphs where previously there was just a single varnode().

Overriding nodes can be useful for unit testing. If you have a node that you don’t want to evaluate as part of a unit test it can be overriden with a mock node.

Shifted Contexts¶

As mentioned above, a node only has a value in a context. In some situations it’s useful to be able to evaluate a node in a context given another node is set to some value without modifying the context.

Consider the following DAG

Suppose you want to evaluate A for all x in [1, 2, 3, 4, 5] but you don’t want to actually affect any other values in the context. You could do that by shifting the context and evaluating A on the shifted context:

from mdf import MDFContext, evalnode, varnode
from datetime import datetime

x = varnode()

@evalnode
def A():
    return x() * 2

@evalnode
def B():
    return x() * 3

ctx = MDFContext()

# set some value for 'x'
ctx[x] = 100

print ctx[A] # 200
print ctx[B] # 300

# calculate A[x=1,2,3,4,5] without modifying the context
for i in [1, 2, 3, 4, 5]:
    shifted_ctx = ctx.shift({x : i})
    print shifted_ctx[A] # 2, 4, 6, 8, 10

print ctx[A] # nothing's changed, still 200
print ctx[B] # nothing's changed, still 300

Shifting a context creates a new context with the shifted value set to a new value, but the shifted context is linked to the original context.

All values not dependent on the shifted value are still shared between the contexts. If you change one in one context it changes in all related contexts. This also means that the cached calculated values are also shared and so shifting a context can be more efficient than simply cloning it.

mdf provides a function shift() for use within a node function. It returns the values of a node in multiple shifted contexts and can be used to create new sub-graphs in the DAG, for example:

This relatively complicated looking DAG can be written by shifting A by x:

from mdf import varnode, evalnode, shift

x = varnode()

@evalnode
def A():
    return x() * 2

@evalnode
def sum_of_A():
    all_As = shift(A, x, [1,2,3,4,5])
    return sum(all_As)

This allows for code that behaves like sub-routines and loops but retains the DAG struture.

Shifting works with overriding nodes also. If the shift value is actually a node instance then the shifted node will be overriden by that node in the shifted context.

Back-Testing¶

In the context of mdf, a back test is nothing more than evaluating a node or collection of nodes for a range of dates. Its possible to do this with a simple for loop:

import mdf
import pandas as pa
from datetime import datetime

# non-existent module used just for illustration in this example
import example_nodes

# get the range of dates used for the back test
date_range = pa.DateRange(datetime(2000, 1, 1), datetime(2011, 9, 6))

# create the context and set some initial values
ctx = mdf.MDFContext()
ctx[example_nodes.an_example_var_node] = 5

# evaluate a node for every date in date_range
results = []
for date in date_range:
    ctx.set_date(date)
    value = ctx[example_nodes.an_example_evalnode]
    results.append(value)

# now we can create a dataframe or plot the values etc.

To simplify this mdf provides several higher level functions that do this for you:

# evaluate a node and return a dataframe of the results
# the column name is the node name
df = mdf.build_dataframe(date_range, example_nodes.an_example_evalnode)

# or supply a list of nodes
df = mdf.build_dataframe(date_range,
                                                     [example_nodes.an_example_evalnode,
                              example_nodes.another_example_evalnode]
                             ctx=ctx)

# plot is the same as build_dataframe but it plots the results
mdf.plot(date_range,
         example_nodes.an_example_evalnode,
         ctx=ctx)

# to_csv is the same as build_dataframe but it writes the results to a csv file
fh = open("myfile.csv")
df = mdf.to_csv(fh,
                date_range,
                example_nodes.an_example_evalnode,
                ctx=ctx)

# get_final_values steps through all the dates in the range but only returns
# the final result
value = mdf.get_final_values(date_range,
                                 example_nodes.an_example_evalnode,
                                 ctx=ctx)

values = mdf.get_final_values(date_range,
                                  [example_nodes.an_example_evalnode,
                                   example_nodes.another_example_evalnode],
                                  ctx=ctx)

# DataFrameBuilder can be used for collecting values into a dataframe
df_builder = mdf.DataFrameBuilder([example_nodes.an_example_evalnode,
                                       example_nodes.another_example_evalnode])

# CSVWriter can be used for writing values to a csv file
csv_builder = mdf.CSVWriter("myfile.csv",
                                [example_nodes.an_example_evalnode,
                                example_nodes.another_example_evalnode])

# or you can use custom functions as well
def my_func(ctx):
    print ctx.get_date(), ctx[example_nodes.an_example_evalnode]

# they're all run in one go using run
mdf.run(date_range, [df_builder, csv_builder, my_func], ctx=ctx)
    # you can then get the dataframe
    df = df_builder.get_dataframe(ctx)

Scenario Analysis¶

As calculations done using mdf are constructed as a DAG all the inputs are accessible, and this lends itself very convieniently to doing scenario analysis.

Using mdf it’s possible to run a back-test for a date range with multiple sets of input parameters simultaneously. By using shifted contexts to achieve this anything not depedendent on the input parameters being varied intermediate calculations can be shared, potentially making the overall run-time less than if the code was run N times with the different input paramters.

Rather than creating all the shifted contexts and iterating through the date range each time you want to run a scenario the run() function may be used:

# calculate example_nodes.an_example_evalnode for a range of different
# values for example_nodes.an_example_var_node

# each scenario is specified as a 'shift' dictionary which is a dictionary
# of nodes to shifted values. In this case only one node is shifted but it
# could be multiple.

shifts = [
    {example_nodes.an_example_var_node : 1},
    {example_nodes.an_example_var_node : 2},
    {example_nodes.an_example_var_node : 3},
    {example_nodes.an_example_var_node : 4},
]

# create a dataframe builder to collect the results of the scenarios
df_builder = mdf.DataFrameBuilder(example_nodes.an_example_evalnode)

# run all the scenarios
mdf.run(date_range, [df_builder], shifts=shifts, ctx=ctx)

# df_builder.dataframes is now a list of dataframes, one for each shift set

# artificial example just for illustration
a = varnode()
b = varnode()

@evalnode
def X():
    return a() + b()

# calculate the value of X for a in [1, 2, 3, 4] and b in [10, 20, 30, 40]
a_values = [1, 2, 3, 4]
b_values = [10, 20, 30, 40]

results = mdf.scenario(date_range,
                            X,
                            a, a_values,
                            b, b_values,
                            ctx=ctx)

# plots the results of each shift as a 3d surface
results = mdf.plot_surface(date_range,
                               X,
                               a, a_values,
                               b, b_values,
                               ctx=ctx)

# plots the results of each shift as a heatmap
results = mdf.heatmap(date_range,
                          X,
                          a, a_values,
                          b, b_values,
                          ctx=ctx)

Working with Timeseries¶

Nodes can be evaluated over a time range to produce time series of results, which can be plotted, stored as pandas DataFrames or exported to Excel.

The main functions used to construct timeseries of results are:

• %mdf_df for creating dataframes
• %mdf_plot for plotting using matplotlib
• %mdf_xl for exporting to Excel

All these functions take two dates (start and end) followed by a list of nodes (note the use of T as a shortcut for today)

In [11]: %mdf_plot 2005-01-01 T rand

Nodes can be defined interactively either by writing new functions or simply using the nodetype method syntax (nodetype_method_syntax) to build up series of operation quickly

In [12]: %mdf_plot 2005-01-01 T rand.cumprodnode() rand.nansumnode()

This could also be written as

In [13]: a = rand.cumprodnode()

In [14]: b = rand.nansumnode()

In [15]: %mdf_plot 2005-01-01 T a b

In addition to these functions there is also %mdf_dfs which returns a list of dataframes (one for each node) and %mdf_wp which returns a widepanel constructed from dataframes for each node. These can be useful when evaluating multiple nodes at the same time but when you don’t want the results to get merged into a single dataframe.

Working with Data¶

Most often data is loaded as a pandas DataFrame, Series or WidePanel. To use those effectively in mdf we normally define a node that is the ‘current’ row or item from that dataset. As time advances that node updates to reveal the data from the underlying structure.

The datanode() function can be used to construct such a node from any DataFrame, WidePanel or series that is indexed by date.

The following example shows how to access data in a pandas DataFrame:

    from mdf import datanode
    import pandas as pa

# load some data
df = pa.DataFrame.from_csv("data_file.csv")

# create a node whose value is the row from the dataframe for 'now'
df_node = datanode("x", df)

This df_node node is like any other mdf node

In [18]: %mdf_plot 2000-01-01 T df_node

Applying Functions¶

Usually when writing code using mdf new nodes are written whenever a value derived from other nodes is required

@evalnode
def df_node_sq():
    return math.pow(df_node(), 2)

For trivial functions such as the one above it can be inconvenient to have to write these for each desired node.

The applynode() function can be used to create new nodes that apply a function to other nodes. The above example can be re-written as follows:

In [19]: df_node_sq = applynode(math.pow, df_node, 2)

In [20]: %mdf_plot 2000-01-01 T df_node_sq

Accessing the Context and Shifting¶

When you want to evaluate a node with another node set to a specific value or overriden you use a shifted context (see Shifted Contexts).

You can get and set the current ambient context using the %mdf_ctx magic function. This allows you to get the current context, create a shifted context and then set that shifted context as the current context.

In [21]: ctx = %mdf_ctx

In [22]: x = varnode(default=1)

In [23]: shifted_ctx = ctx.shift({x : 2})

In [24]: %mdf_ctx shifted_ctx
Out[24]: <ctx 1: 2012-05-03 [x=2] at 123373200>

In [25]: x()
Out[25]: 2

The functions %mdf_df, %mdf_plot and %mdf_xl also take an optional set of shifts. This makes it easy to get results from a shift without having the get, shift and set the current context.

In [26]: df_node_pow = applynode(math.pow, df_node, x)

In [27]: %mdf_plot 2000-01-01 T df_node_pow [x=0.5]

Using the MDF Viewer¶

The mdf viewer can be used explore the dependencies between nodes and plot or export values over time.

The mdf viewer can be opened from ipython with the magic command %mdf_show.

In [28]: %mdf_show df_node_pow

More nodes can be added to the open viewer using the same command.

In [29]: %mdf_show rand

To plot or export nodes select the nodes you want (use Shift or Ctrl to select multiple nodes) and then right click and select plot from the context menu. The same context menu may be used to export values to Excel or to render a graphical representation of the graph (requires Graphviz to be installed).

Once the viewer is open you can select one or more nodes and then use the magic command %mdf_selected to get your current selection in your IPython session

In [29]: %mdf_selected
Out[29]:
[(<ctx 0: 2005-02-11 at 115387248>,
  <<type 'mdf.nodes.MDFEvalNode'> [name=rand] at 0x30fee48>)]

This returns a list of contexts and node objects that correspond to what is selected in the viewer.

The magic functions %mdf_plot, %mdf_df and %mdf_xl can also be used to plot or get results for the currently selected nodes. To use the currently selected nodes don’t specify any nodes at all on the command line.

%mdf_plot 2005-01-01 T

Nodes Types¶

mdf.varnode([name] [, default] [, category])¶

Creates a simple MDFNode that can have a value assigned to it in a context.

It may also take a default value that will be used if no specific value is set for the node in a context.

A varnode may be explicitly named using the name argument, or if left as None the variable name the node is being assigned to will be used.

my_varnode = varnode(default=100)

mdf.evalnode(func [, filter] [, category])¶

Decorator for creating an MDFNode whose value is determined by calling the function func.

func should be a function or generator that takes no arguments and returns or yields the current value of the node.

If func is a generator instead of a function it will be advanced as the now() node is advanced. This can be used to calculate accumulated values and maintain internal state over evaluations:

@evalnode
def some_function():
    # Set initial value of 'accum' to 0
    accum = 0

    # yield the initial value of 'accum' (0 in this case)
    yield accum

    while True:
        accum += 1
        # yield the updated value of 'accum' on each evaluation
        yield accum

Yield essentially bookmarks the current execution point, and returns the supplied value (accum in the above example). When the node is evaluated again, execution will resume at the bookmark and continue until the next yield statement is encountered. By using an infinite while loop the node can be evaluated any number of times.

The above example can actually be shortened to:

@evalnode
def some_function():
    # Set initial value of 'accum' to 0
    accum = 0

    while True:
        yield accum
        accum += 1

In this case, yield will first return the initial value of ‘accum’. On subsequent evaluations, the incrementation step will be also executed and the node will produce the updated value.

filter may be used in the case when func is a generator to prevent the node valuation being advanced on every timestep. If supplied, it should be a function or node that returns True if the node should be advanced for the current timestep or False otherwise.

mdf.queuenode(func [, size] [, filter] [, category])¶

Decorator for creating an MDFNode that accumulates values in a collections.deque each time the context’s date is advanced.

The values that are accumulated are the results of the function func. func is a node function and takes no arguments.

If size is specified the queue will grow to a maximum of that size and then values will be dropped off the queue (FIFO).

size may either be a value or a callable (i.e. a function or a node):

@queuenode(size=10)
def node():
    return x

or:

# could be an evalnode also
queue_size = varnode("queue_size", 10)

@queunode(size=queue_size)
def node():
    return x

or using the nodetype method syntax (see nodetype_method_syntax):

@evalnode
def some_value():
    return ...

@evalnode
def node():
    return some_value.queue(size=5)

mdf.delaynode(func [, periods] [, initial_value] [, lazy] [, filter] [, category])¶

Decorator for creating an MDFNode that delays values for a number of periods corresponding to each time the context’s date is advanced.

The values that are delayed are the results of the function func. func is a node function and takes no arguments.

periods is the number of timesteps to delay the value by.

initial_value is the value of the node to be used before the specified number of periods have elapsed.

periods, initial_value and filter can either be values or callable objects (e.g. a node or a function):

@delaynode(periods=5)
def node():
    return x

or:

# could be an evalnode also
periods = varnode("periods", 5)

@delaynode(periods=periods)
def node():
    return x

If lazy is True the node value is calculated after any calling nodes have returned. This allows nodes to call delayed version of themselves without ending up in infinite recursion.

The default for lazy is False as in most cases it’s not necessary and can cause problems because the dependencies aren’t all discovered when the node is first evaluated.

e.g.:

@delaynode(periods=10)
def node():
    return some_value

or using the nodetype method syntax (see nodetype_method_syntax):

@evalnode
def some_value():
    return ...

@evalnode
def node():
    return some_value.delay(periods=5)

mdf.nansumnode(func [, filter] [, category])¶

Decorator that creates an MDFNode that maintains the nansum of the result of func.

Each time the context’s date is advanced the value of this node is calculated as the nansum of the previous value and the new value returned by func.

e.g.:

@nansumnode
def node():
    return some_value

or using the nodetype method syntax (see nodetype_method_syntax):

@evalnode
def some_value():
    return ...

@evalnode
def node():
    return some_value.nansum()

mdf.cumprodnode(func [, filter] [, category])¶

Decorator that creates an MDFNode that maintains the cumulative product of the result of func.

Each time the context’s date is advanced the value of this node is calculated as the previous value muliplied by the new value returned by func.

e.g.:

@cumprodnode
def node():
    return some_value

or using the nodetype method syntax (see nodetype_method_syntax):

@evalnode
def some_value():
    return ...

@evalnode
def node():
    return some_value.cumprod()

TODO: That node needs a test for the argument skipna, since it is not entirely clear what it should do if the first value is na. It would be nice to be able to specify an initial value.

mdf.ffillnode(func[, initial_value])¶

Decorator that creates an MDFNode that returns the current result of the decoratored function forward filled from the previous value where the current value is NaN.

The decorated function may return a float, pandas Series or numpy array.

e.g.:

@ffillnode
def node():
    return some_value

or using the nodetype method syntax (see nodetype_method_syntax):

@evalnode
def some_value():
    return ...

@evalnode
def node():
    return some_value.ffill()

mdf.rowiternode(func [, index_node=now] [, missing_value=np.nan] [, filter] [, category])¶

Decorator that creates an MDFNode that returns the current row of item of a pandas DataFrame, WidePanel or Series returned by the decoratored function.

What row is considered current depends on the index_node parameter, which by default is now.

missing_value may be specified as the value to use when the index_node isn’t included in the data’s index. The default is NaN.

delay can be a number of timesteps to delay the index_node by, effectively shifting the data.

ffill causes the value to get forward filled if True, default is False.

e.g.:

@rowiternode
def datarow_node():
    # construct a dataframe indexed by date
    return a_dataframe

@evalnode
def another_node():
    # the rowiternode returns the row from the dataframe
    # for the current date 'now'
    current_row = datarow_node()

or using the nodetype method syntax (see nodetype_method_syntax):

@evalnode
def dataframe_node():
    # construct a dataframe indexed by date
    return a_dataframe

@evalnode
def another_node():
    # get the row from dataframe_node for the current_date 'now'
    current_row = dataframe_node.rowiter()

mdf.returnsnode(func [, filter] [, category])¶

Decorator that creates an MDFNode that returns the returns of a price series.

NaN prices are filled forward. If there is a NaN price at the beginning of the series, we set the return to zero. The decorated function may return a float, pandas Series or numpy array.

e.g.:

@returnsnode
def node():
    return some_price

or using the nodetype method syntax (see nodetype_method_syntax):

@evalnode
def some_price():
    return ...

@evalnode
def node():
    return some_price.returns()

The value at any timestep is the return for that timestep, so the methods ideally would be called ‘return’, but that’s a keyword and so returns is used.

mdf.applynode(func, [, args=()] [, kwargs={}] [, category])¶

Return a new mdf node that applies func to the value of the node that is passed in. Extra args and kwargs can be passed in as values or nodes.

Unlike most other node types this shouldn’t be used as a decorator, but instead should only be used via the method syntax for node types, (see nodetype_method_syntax) e.g.:

A_plus_B_node = A.applynode(operator.add, args=(B,))

mdf.lookaheadnode(func, periods [, offset=pa.datetools.BDay()] [, filter] [, category])¶

Node type that creates an MDFNode that returns a pandas Series of values of the underlying node for a sequence of dates in the future.

Unlike most other node types this shouldn’t be used as a decorator, but instead should only be used via the method syntax for node types, (see nodetype_method_syntax) e.g.:

future_values = some_node.lookahead(periods=10)

This would get the next 10 values of some_node after the current date. Once evaluated it won’t be re-evaluated as time moves forwards; it’s always the first set of future observations. It is intended to be used sparingly for seeding moving average calculations or other calculations that need some initial value based on the first few samples of another node.

The dates start with the current context date (i.e. now()) and is incremented by the optional argument offset which defaults to weekdays (see pandas.datetools.BDay).

Parameters:	periods (int) – the total number of observations to collect, excluding any that are ignored due to any filter being used. offset – date offset object (e.g. datetime timedelta or pandas date offset) to use to increment the date for each sample point. filter – optional node that if specified should evaluate to True if an observation is to be included, or False otherwise.

Node Factory Functions¶

mdf.datanode([name=None,] data [, index_node] [, missing_value] [, delay] [, name] [,filter] [,category])¶

Return a new mdf node for iterating over a dataframe, panel or series.

data is indexed by another node index_node, (default is now()), which can be any node that evaluates to a value that can be used to index into data.

If the index_node evaluates to a value that is not present in the index of the data then missing_value is returned.

missing_value can be a scalar, in which case it will be converted to the same row format used by the data object with the same value for all items.

delay can be a number of timesteps to delay the index_node by, effectively shifting the data.

ffill causes the value to get forward filled if True, default is False.

data may either be a data object itself (DataFrame, WidePanel or Series) or a node that evaluates to one of those types.

e.g.:

df = pa.DataFrame({"A" : range(100)}, index=date_range)
df_node = datanode(data=df)

ctx[df_node] # returns the row from df where df == ctx[now]

A datanode may be explicitly named using the name argument, or if left as None the variable name the node is being assigned to will be used.

mdf.filternode([name=None,] data [, index_node] [, delay] [, name] [,filter] [,category])¶

Return a new mdf node for using as a filter for other nodes based on the index of the data object passed in (DataFrame, Series or WidePanel).

The node value is True when the index_node (default=now) is in the index of the data, and False otherwise.

This can be used to easily filter other nodes so that they operate at the same frequency of the underlying data.

delay can be a number of timesteps to delay the index_node by, effectively shifting the data.

A filternode may be explicitly named using the name argument, or if left as None the variable name the node is being assigned to will be used.

Custom Node Types¶

mdf.nodetype(func)¶

decorator for creating a custom node type:

#
# create a new node type 'new_node_type'
#
@nodetype
def new_node_type(value, fast, slow):
    return (value + fast) * slow

#
# use the new type to create a node
#
@new_node_type(fast=1, slow=10)
def my_node():
    return some_value

# ctx[my_node] returns new_node_type(value=my_node(), fast=1, slow=10)

The node type function takes the value of the decorated node and any other keyword arguments that may be supplied when the node is created.

The node type function may be a plain function, in which case it is simply called for every evaluation of the node, or it may be a co-routine in which case it is sent the new value for each iteration:

@nodetype
def nansumnode(value):
    accum = 0.
    while True:
        accum = np.nansum([value, accum])
        value = yield accum

@nansumnode
def my_nansum_node():
    return some_value

The kwargs passed to the node decorator may be values (as shown above) or nodes which will be evaluated before the node type function is called.

Nodes defined using the @nodetype decorator may be applied to classmethods as well as functions and also support the standard node kwargs ‘filter’ and ‘category’.

Node types may also be used to add methods to the MDFNode class (See nodetype_method_syntax):

@nodetype(method="my_nodetype_method")
def my_nodetype(value, scale=1):
    return value * scale

@evalnode
def x():
    return ...

@my_nodetype(scale=10)
def y():
    return x()

# can be re-written as:
y = x.my_nodetype_method(scale=10)

Pre-defined Nodes¶

mdf.now()¶

Pre-defined node present in every context that always evaluates to the date set on the context.

See MDFContext.get_date() and MDFContext.set_date().

Functions¶

mdf.shift(node, target [, values] [, shift_sets])¶

This function is for use inside node functions.

Applies shifts to the current context for each shift specified and returns the value of ‘node’ with each of the shifts applied.

If target and values are specified ‘target’ is a node to apply a series of shifts to, specified by ‘values’.

If shifts_sets is specified, ‘shift_sets’ is a list of nodes to values dictionaries, each one specifying a shift.

If the same shift set dictionaries are used several times ShiftSet objects may be used instead which will be slightly faster. See make_shift_set().

Returns a list of the results of evaluating node for each of the shifted contexts in the same order as values or shift_sets.

See MDFContext.shift() for more details about shifted contexts.

mdf.run(date_range [, callbacks=[]] [, values={}] [, shifts=None] [, filter=None] [, ctx=None])¶

creates a context and iterates through the dates in the date range updating the context and calling the callbacks for each date.

If the context needs some initial values set they can be passed in the values dict or as kwargs.

For running the same calculation but with different inputs shifts can be set to a list of dictionaries of (node -> value) shifts.

If shifts is not None and num_processes is greater than 0 then that many child processes will be spawned and the shifts will be processed in parallel.

Any time-dependent nodes are reset before starting by setting the context’s date to datetime.min (after applying time zone information if available).

mdf.plot(date_range, nodes [, labels=None] [, values={}] [, filter=None] [, ctx=None])¶

evaluates a list of nodes for each date in date_range and plots the results using matplotlib.

mdf.build_dataframe(date_range, nodes [, labels=None] [, values={}] [, filter=None] [, ctx=None])¶

evaluates a list of nodes for each date in date_range and returns a dataframe of results

mdf.get_final_values(date_range, nodes [, labels=None] [, values={}] [, filter=None] [, ctx=None])¶

evaluates a list of nodes for each date in date_range and returns a list of final values in the same order as nodes.

mdf.scenario(date_range, result_node, x_node, x_shifts, y_node, y_shifts [, values={}] [, filter=None] [, ctx=None] [, dtype=float])¶

evaluates a single result_node for each date in date_range and gets its final value for each shift in x_shifts and y_shifts.

x_shifts and y_shifts are values for x_node and y_node respectively.

result_node should evaluate to a single float, and the result is a 2d nparray

mdf.plot_surface(date_range, result_node, x_node, x_shifts, y_node, y_shifts [, values={}] [, filter=None] [, ctx=None] [, dtype=float])¶

evaluates a single result_node for each date in date_range and gets its final value for each shift in x_shifts and y_shifts.

x_shifts and y_shifts are values for x_node and y_node respectively.

result_node should evaluate to a single float.

The results are plotted as a 3d graph and returned as a 2d numpy array.

mdf.make_shift_set(shift_set_dict)¶

Return a ‘ShiftSet’ object that encapsulates the information required to get a shifted context.

This can be used to pass to the shift() function instead of a dictionary for better performance when regularly shifting by the same thing.

Classes¶

Building MDF¶

python setup.py build_ext --inplace

python setup.py bdist_egg

Source Code Overview¶

a_var_node = varnode()

Node Evaluation¶

Nodes only have values in a context, so it makes sense that to get a nodes value it’s evaluated by starting with the context. Indexing into the context with a node (see MDFContext.__getitem__()) calls MDFContext.get_value(), which is a public API method. This calls an internal C method MDFContext._get_node_value() which is where the work is actually done.

Once inside a node evaluation to avoid passing the current context around to every node call to allow the nodes to evaluate other nodes, there’s the notion of a currently active context. The currently active context is stored in a dictionary that maps thread id to the MDFContext (_current_contexts in context.py). See also _get_current_context().

# commutative
ctx.shift({a: x}).shift({b: y}) == ctx.shift({b: y}).shift({a: x})

# associative
ctx.shift({a: x, b: y}).shift({c: z}) == ctx.shift({a: x}).shift({b: y, c: z})

a = varnode()
b = varnode()

@evalnode
def foo():
    return a()


ctx = MDFContext()
shifted_a = ctx.shift({a : 1})
shifted_b = shifted_a.shift({b : 2})

shifted_b[foo] == shifted_a[foo]

Node Types¶

Other node types (e.g. queuenode(), ffillnode()) are built on top of MDFEvalNode. The basic concept is that a nodetype does the job of a normal evalnode but then calls a second function on the result of that to transform it in some way.

The second function that does that transformation is referred to as the node type function and the inner function that gets evaluated to provide the input to the node type function is the node function.

Here’s an exampple of a very simple custom node type that will help illustrate how node types work:

@nodetype
def node_type_function(value):
    return value * 2

@node_type_function
def node_function():
    return x

When evaluating the node node_function the order of execution is to first call the python function node_function and then to pass the result of that to the python function node_type_function. The result of node_type_function becomes the value of the node node_function.

To understand how this works it helps to talk about what types are used and what the result of these decorators is.

nodetype returns an instance of a MDFCustomNodeDecorator. This instance keeps a reference to the decorated function. It’s a callable object and its call method works as a decorator.

So, the decorated node_type_function is an instance of MDFCustomNodeDecorator. When used as a decorator as @node_type_function on another function it returns an instance of MDFCustomNode. This MDFCustomNode is a subclass of MDFEvalNode as is instantiated with the node function and keeps a reference to the node type function. MDFCustomNode differs from MDFEvalNode by calling its node type function after doing the normal evaluation of the node function. The result of this is what gets returned as the final value of the node.

Things get a little more complicated when adding arguments to the node type, for example:

@nodetype
def node_type_function_2(value, multipler):
    return value * multiplier

@node_type_function_2(multiplier=2)
def node_function_2():
    return x

To pass the arguments from the node instantiation (when @node_type_function_2 is called with node_function_2) to the node type function node_type_function_2 the args have to be stored by the MDFCustomNode instance. This is exactly what happens, and when the custom node is evaluated it calls the node type function with these stored arguments.

If any of the arguments are nodes they automatically get evaluated before being passed to the node type function. Occasionally it is necessary to pass nodes in as arguments. MDFCustomNode checks a class property node_kwargs and doesn’t automatically evaluate any args in that list. By subclassing MDFCustomNode this can be set for specific node types.

Class nodes¶

Nodes can be declared as properties of classes as well as modules. These nodes may take a single argument, which will be the class the node is being called on. If they take no arguments they behave the same way as a normal node.

Class nodes have to be aware of the class they’re defined on and the class they’re bound to (accessed from). Consider the following:

class A(object):

    @evalnode
    def foo(cls):
        return "Declared in A, called on %s" % cls.__name__

class B(A):

    @evalnode
    def foo(cls):
        return "Overridden in B (%s)" % super(B, cls).foo()

In one sense this code just declares two nodes, A.foo and B.foo. That’s a slight simplication of what’s actually going on though; if there were only two then super(B, cls).foo() would have to evaluate A.foo() which would return Declared in A, called on A. So, there are actually threee [1] nodes, A:A.foo, B:B.foo() and B:A.foo().

What the code above declares are unbound nodes. That is, nodes that have no knowledge of the classes they belong to. When they are accessed from the class (i.e. A.foo accesses foo from A) they are then bound to the class at that point. Binding creates a new node that includes everything from the original unbound node definition and information about the class the new node is bound to.

MDFEvalNode is a descriptor and so the process of binding nodes to a class is done by MDFEvalNode.__get__(). This is called whenever a node is accessed as a property of a class. All evalnodes have a dictionary of classes to bound versions of themselves. When accessing a node multiple times on the same class the same bound node is returned each time. If no bound node exists for the unbound node and class then a new node is created.

To bind the node to a class the function that it references must also be bound (to create a staticmethod or classmethod) and the new node is created using that bound method. As keyword arguments to the node may also reference unbound functions or nodes those too need to be bound. MDFEvalNode._bind() handles binding any additional functions or nodes and may be overridden by any subclasses requiring additional objects to also be bound. The helper method MDFEvalNode._bind_function is used to create bound versions of individual functions, methods and other callable object types.