The Foldable type class captures the concept of data structures that we can iterate over. Lists are foldable, as are Vectors and Streams.

Foldables give us a way to process values embedded in a structure as if they existed in a sequential order, as we’ve seen with list folding.

Foldable gives us great use cases for monoids and the Eval monad.

In general, a fold function allows users to transform one algebraic data type to another.

The typical use case is to accumulate a value as we traverse. We supply an accumulator value and a binary function to combine it with an item in the sequence.

The function produces another accumulator, allowing us to recurse down the sequence. When we reach the end, the final accumulator is our result.

In addition to these familiar methods, Cats provides two methods that make use of monoids:

fold (and its alias combineAll) combines all elements in the sequence using their Monoid foldMap maps a user-supplied function over the sequence and combines the results using a Monoid

The interesting thing about fold and foldMap is that they use a Monoid instead of a function to give us the final result.

One very important aspect to understand here is that it is the fold function that requires the elements of Foldable to have a Monoid instance, while Foldable itself does not have that restriction.

In functional programming it is very common to encode "effects" as data types - common effects include Option for possibly missing values, Xor and Validated for possible errors, and Future for asynchronous computations.

These effects tend to show up in functions working on a single piece of data - for instance parsing a single String into an Int, validating a login, or asynchronously fetching website information for a user.

Each function asks only for the data it actually needs; in the case of userInfo, a single User.

We could write a function that takes a list of users and fetches profiles for all of them, but that would be a bit strange.

If we just wanted to fetch the profile of just one user, we would either have to wrap it in a List or write a separate function that takes in a single user anyways.

Note the return type List[Future[Profile]]. This makes sense given the type signatures, but seems unwieldy.

We now have a list of asynchronous values, and to work with those values we must then use the combinators on Future for every single one.

It would be nicer instead if we could get the aggregate result in a single Future, say a Future[List[Profile]].

As it turns out, the Future companion object has a traverse method on it.

However, that method is specialized to standard library collections and Futures - there exists a much more generalized form that would allow us to do things like parse a List[String] or validate credentials for a List[User].

Enter Traverse.

sequence threads all the G effects through the F structure to invert the structure from F[G[_]] to G[F[_]].

traverse allows you to transform elements inside the structure like a Functor, producing applicative effects along the way, and lift those instances of applicative structure outside of the Traversable structure.

Given a function which returns a G effect, traverse threads this effect through the running of this function on all the values in F, returning an F[A] in a G context.

In our above examples, F is List, and G is Option, Xor, or Future.

For the profile example, traverse says given a List[User] and a function User => Future[Profile], it can give you a Future[List[Profile]].

More generally, F[_] is some sort of context which may contain a value (or several). While List tends to be among the most general cases, there also exist Traversable instances for Option, Xor, and Validated (among others).

We’re still mapping a function over some embedded value(s), like map, but similar to flatMap, the function is itself generating more structure.

However, unlike flatMap, the generated structure is of a different type than the embedded structure.

Finally, note the implied relationship between Foldable and Traverse.

• Foldable does not extend Monoid, but has methods that rely on monoidal values (e.g. def fold[A: Monoid](fa: F[A]))

• Traverse does not extend Applicative, but has methods that rely on applicative structures (e.g. def traverse[G[_]: Applicative, A, B](fa: F[A])(f: A => G[B]))

Traverse[T[_]] has two laws.

There are many ways to state the laws, we present one way here, but you may see them in different configurations elsewhere.

The tutorial contains an exercise on combining two monads into one composite monad when the inner monad has a Traverse instance.

While the types always allow this, the result fails to meet the monad laws unless the traversable monad is also a commutative monad.

Two examples of commutative monads are Reader and Option.

Two examples of monads that are not commutative are List and State.

See the Wikipedia article on the "distributive law between monads" and a more in-depth article on n-lab.

Going back to our earlier Future example, we can write an Applicative instance for Future that runs each Future concurrently.

Then when we traverse a List[A] with an A => Future[B], we perform a scatter-gather operation

Each A creates a concurrent computation that will produce a B (the scatter), and as the futures complete they will be gathered back into a list.

Another interesting effect we can use is Reader. Recall that a Reader[E, A] is a type alias for Kleisli[Id, E, A] which is a wrapper around E => A.

If we fix E to be some sort of environment or configuration, we can use the Reader applicative in our traverse.

When our new job's run method is called, it will go through each topic and run its topic-specific job, collecting results as it goes.

Note that our job's run method has type signature Context => Result, so it requires a Context before producing the value we want.

For example, in Spark the information needed to run a Spark job (where the master node is, memory allocated, etc.) resides in a SparkContext.

Going back to the above example, we can see how one may define topic-specific Spark jobs (type Job[A] = Reader[SparkContext, A]) and then run several Spark jobs on a collection of topics via traverse.

We then get back a Job[List[Result]], which is equivalent to SparkContext => List[Result].

When finally passed a SparkContext, we can run the job and get our results back.

Moreover, the fact that our aggregate job is not tied to any specific SparkContext allows us to pass in a SparkContext pointing to a production cluster, or (using the exact same job) pass in a test SparkContext that just runs locally across threads. This makes testing large jobs easy.

Finally, this encoding ensures that all the jobs for each topic run on the exact same cluster. At no point do we manually pass in or thread a SparkContext through - that is taken care for us by the (applicative) effect of Reader and therefore by traverse.

The Essence of the Iterator Pattern, by Jeremy Gibbons and Bruno Oliveira. Published in Mathematically-Structured Functional Programming, 2006.

An Investigation of the Laws of Traversals, by Mauro Jaskelioff and Ondrej Rypacek, published in Mathematically-Structured Functional Programming, 2012.

Typeclassopedia is a great resource on Monad, Applicative, Traverse and other type classes.