CCES Unicamp

ChunkSplitters.jl: easy split the elements or indices of a collection into chunks.

https://juliafolds2.github.io/ChunkSplitters.jl/stable/#ChunkSplitters.jl

ChunkSplitters.jl makes it easy to split the elements or indices of a collection into chunks:

julia> using ChunkSplitters

julia> x = [1.2, 3.4, 5.6, 7.8, 9.1, 10.11, 11.12];

julia> for inds in index_chunks(x; n=3)
           @show inds
       end
inds = 1:3
inds = 4:5
inds = 6:7

julia> for c in chunks(x; n=3)
           @show c
       end
c = [1.2, 3.4, 5.6]
c = [7.8, 9.1]
c = [10.11, 11.12]

This can be useful in many areas, one of which is multithreading, where we can use chunking to control the number of spawned tasks:

function parallel_sum(x; ntasks=nthreads())
    tasks = map(chunks(x; n=ntasks)) do chunk_of_x
        @spawn sum(chunk_of_x)
    end
    return sum(fetch, tasks)
end

Working with chunks and their respective indices also improves thread-safety compared to a naive parallelisation approach based on threadid(). 

 

Related posts

MDLovoFit: Automatic identification of mobile and rigid substructure in protein dynamics

Leandro Martinez

UNICAMP engineers are designing models to simulate how ground vibrations affect Sirius

escience

Converting scripts into reproducible Workflow Research Objects

escience