---
title: "reference-book/4-dataflows/collector"
source: reference-book
version: genesis-0-1-0
id: reference-book/4-dataflows/collector
canonical: /docs/reference-book/4-dataflows/collector
---


Collector operator <<<..>>> is used when data batching is required. As you may have noticed simorg doesn't have the familiar concept of <<<Array>>> similar to CPLs.

The reason is simple, we don't want to interrupt and reshape the structure of data as much as possible. While data is moving through our system it can temporarily be accumulated in a point to fulfill certain requirements.

Unlike CPLs that store the data in memory and then process it, Simorg promotes the idea of stream processing. So the best possible solution is to process data as part of natural dataflow and avoid storing it unless there is a certain business logic. One possible example is the logic that requires data to be stored in memory (temporarily) until a certain event happens, allowing it to move forward.

In these scenarios <<<..>>> becomes quite handy. You can imagine it as a water tank which acts as a temporary storage for water in a distribution system.

Let's see a simple example of this operator,

```
:[0, :] $data <10 ++data

data ..10 ?

```

If you run the code you will see the following log in the console:

```
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
```
The code is simple, line 1 generates a self vibrating variable that generates 10 vibrations and on line 3, each of those vibrations is captured inside our collector until it reaches the capacity so releasing its vibration.

<<<{ }>>> wrapper is used to mark collectors in simorg. A <<<collection>>> is equivalent to an <<<array>>> as data is stored inside it based on the order it is being received. But we avoid calling it an <<<array>>> in order not to create the same expectation of an <<<array>>> like in a CPL. Instead we call it a <<<collection>>>.

It's not possible to access the data inside a <<<collection>>>, for example, using an <<<index>>> mechanism as there is no <<<index>>>. Iterating over data is possible by using another collector operator which is acting similar to a destructor. Let's continue our previous example by adding another collector in series to the first one.

```
:[0, :] $data <10 ++data
data ..10..2 ?

```

Here we are using two collectors back to back in order to first batch the data and then destruct the batch and create a stream of data one by one.

So first collector is not vibrating until its capacity has been reached and it has gathered <<<3>>> elements. Then it releases the vibration and the next collector vibrates as soon as it receives one element, meaning there will be three vibrations out of it.

$$$AlertBox type=HINT title=Do Not Collect message=Collectors are temporary placeholders to gather vibrations and release them according to a business logic. Do not see them as arrays. Data should stream as natural as possible. If you need to apply a logic on each single vibration, then do it naturally while the vibrations are streaming.$$$

This is a powerful mechanism to batch and then process in parallel. As we will introduce in future sections, dataflow pipelines are quite powerful and smart to pick the best processing model for incoming data.

For example in this case, the engine is smart enough to perform all these possible tasks in one <<<tick>>>. So there will be three different vibrations after the second collector but all of them in one <<<tick>>> which is quite efficient. We will talk about the concept of <<<tick>>> in future chapters but for now consider a <<<tick>>> a unit of processing.

If the operand is not a <<<Natural Number>>> (1,2,3,...), it will act as a trigger. So collector will continue collecting incoming vibrations until the operand is vibrated.

This operator is usually used in combination with other operators to fulfill complex logic.

To insist and emphasize more, simorg is promoting natural flow of data. According to this pattern the usage of <<<index>>> in designing a solution should not have any place. Data itself should have enough characteristics to replace the necessity of having an <<<index>>>.
But still for some edge cases, it is possible to generate an index and attach it to incoming data and then push it into a collector.
This is not an anti-pattern to what we mentioned earlier as now the data itself will have a field acting as an index. Still one should really be careful with this kind of implementation as data itself is the final actor, it should have enough facts inside it to overcome the necessity of an <<<index>>>.

This is quite similar to the concept of water flowing in a pipe or electrons flowing in a wire, we don't care about each specific molecule of water or each single electron but their act of being itself in a place is enough to fulfill their duty. Same is true about the stream of data (bytes), flowing inside Simorg's engine.

The topic of <<<collection>>> length or size also can be seen as part of the same reasoning. Actually, instead of checking the length and size later, all the business logic about length and size should be taken care of in parallel as the collector is filling up. Let's not forget that these are vibrations that are filling up the collector.
It's all about natural flow of data. Using this approach, data will be where it needs to be without babysitting it and manually managing it.

Long story short, don't expect to fill up the collection and then try to know the size of it. Whatever the size dependent logic is, it should naturally happen in parallel when the collector is filling up.

$$$AlertBox type=HINT title=Why Keeping Distance From Arrays? message= It is a valid question, the biggest problem with arrays is that they force us to embrace a sequential processing model which is no longer a modern way of processing. Decisions like this are rooted in a strategic model which is embracing mathematical data structures and parallel processing.$$$

We will see examples of size logic with collectors in Common Patterns chapter.

A collection follows the definition of an <<<OR>>> gate for the <<<truthy>>> check. If at least one element is <<<truthy>>>, the whole collection is considered <<<truthy>>>.


Certain functionalities e.g. sorting will be supported using the <<<array>>> plugin from simorg's stl.