dds.core package

Submodules

dds.core.partition_generators module

PyCOMPSs DDS - Partition generators.

Partitions should be loaded with data in the last step only. Thus, generator objects are necessary. Inside ‘task’ functions we will call their ‘generate’ method in order to retrieve the partition. These partitions can previously be loaded on master and sent to workers, or read from files on worker nodes.

class dds.core.partition_generators.BasicDataLoader(data)[source]

Bases: IPartitionGenerator

Basic data loader.

retrieve_data()[source]

Retrieve data.

Returns:

Data.

exception dds.core.partition_generators.DDSException(message: str)[source]

Bases: Exception

Generic DDS exception class.

class dds.core.partition_generators.IPartitionGenerator[source]

Bases: object

Everyone implements this.

retrieve_data()[source]

Retrieve data.

Raises:

NotImplementedError – Not implemented function.

class dds.core.partition_generators.IteratorLoader(iterable, start, end)[source]

Bases: IPartitionGenerator

Iterator Loader.

retrieve_data()[source]

Divide and retrieve the next partition.

Returns:

Data.

class dds.core.partition_generators.PickleLoader(pickle_path)[source]

Bases: IPartitionGenerator

Pickle loader.

retrieve_data()[source]

Retrieve data.

Returns:

Data.

class dds.core.partition_generators.WorkerFileLoader(file_paths, single_file=False, start=0, chunk_size=None)[source]

Bases: IPartitionGenerator

Worker file loader.

retrieve_data()[source]

Retrieve data.

Returns:

Data.

dds.core.partition_generators.read_in_chunks(file_name, chunk_size=1024, strip=True)[source]

Lazy function (generator) to read a file piece by piece.

Parameters:

  • file_name – File name to read.

  • chunk_size – Chunk size (Default: 1k).

  • strip – If it requires stripping.

Returns:

Next partition.

dds.core.partition_generators.read_lines(file_name, num_of_lines=1024, strip=True)[source]

Lazy function (generator) to read a file line by line.

Parameters:

  • file_name – File to read.

  • num_of_lines – Total number of lines in each partition.

  • strip – If line separators should be stripped from lines.

Returns:

Next partition.

dds.core.tasks module

PyCOMPSs DDS - Tasks.

dds.core.tasks.distribute_partition(col, func, partitioner_func, partition, collection=None)

Distribute (k, v) structured elements of the partition on ‘buckets’.

Parameters:

  • col – Empty ‘buckets’, must be repleced with COLLECTION_OUT.

  • func – Function from DDS object to be applied to the partition before the distribution.

  • partitioner_func – A function to find element’s corresponding bucket.

  • partition – The partition itself or a partition generator object.

  • collection – If the partition is a collection of future objects.

Returns:

Fill the empty ‘buckets’ with the elements of the partition.

dds.core.tasks.map_and_save_pickle(func, index, path, partition, collection=None)

Map and save pickled file.

Same as ‘map_partition’ function with the only difference that this one saves the result as a pickle file.

Parameters:

  • func – Function to apply.

  • index – Important to keep the order of the partitions.

  • path – Directory to save the partition.

  • partition – Partition.

  • collection – Is collection?

Returns:

None.

dds.core.tasks.map_and_save_text_file(func, index, path, partition, collection=None)

Map and save text file.

Same as ‘map_partition’ function with the only difference that this one saves the result as a text file.

Parameters:

  • func – Function to apply.

  • index – Important to keep the order of the partitions.

  • path – Directory to save the partition.

  • partition – Partition.

  • collection – Is collection?

Returns:

No return value skips the serialization phase.

dds.core.tasks.map_partition(func, partition, collection=None)

Map the given function to the partition.

Parameters:

  • func – A function that return only one argument which is iterable.

  • partition – The partition itself or a partition generator object.

  • collection – Partition when partition is a collection.

Returns:

The transformed partition.

dds.core.tasks.reduce_dicts(first, rest)

Reduce dictionaries.

CAUTION! Modifies first dictionary.

Parameters:

  • first – First dictionary.

  • rest – Second dictionary.

Returns:

None.

dds.core.tasks.reduce_multiple(function, parts)

Reduce multiple.

Parameters:

  • function – Reducing function.

  • parts – List of elements.

Returns:

Reduction result.

dds.core.tasks.task_collect_samples(partition, num_of_samples, key_func)

Collect samples.

Parameters:

  • partition – List of elements.

  • num_of_samples – Number of samples.

  • key_func – Key function.

Returns:

Collected samples.

dds.core.tasks.task_dict_to_list(iterator, total_parts, partition_num)

Convert dictionary to (key, value) pairs.

Parameters:

  • iterator – Iterator object.

  • total_parts – Total parts.

  • partition_num – Number of partitions.

Returns:

List of (key, value) pairs

dds.core.utils module

PyCOMPSs DDS - Utils.

This file contains the DDS interface utilities.

dds.core.utils.default_hash(obj)[source]

Get the hash of the given object.

Parameters:

obj – Object to calculate the hash.

Returns:

Hash value.

Module contents

This package contains the DDS core.