Development - Collector

The Collector module provides mechanisms to manage data consumed by front-end application. To populate the data stores, 4 steps are required:

Touch

In order for anything to be synchronized, first it has to be marked (touched) via the Touch mechanism. Each collector uses this mechanism to be able to determine which data should be added, updated or deleted.


Cascading Updates, depending on the type of data changed, can trigger cascade update of multiple resources.
For example: renaming of a category will result in updating the urls, category nodes, navigation and products.

The basic idea is that, any resource which should be exported, has an entry in the spy_touch table. This table is used by collector’s query, to collect items marked for synchronization.

See Touch for more information.

Touch Record

CollectorQueries must read information from spy_touch table in order to determine which resources should be collected. Depending on Collector Store Type, the spy_touch_storage or spy_touch_search table records are also needed. These records are used to determine the keys of the resources that are exported to client-side data stores, and this combined information represents the Touch Record.

SQL Touch Record Implementation

To write a collector SQL query, you must include the following 3 columns:

  • id_touch from the spy_touch table
  • item_id from the spy_touch table
  • id either from the spy_touch_storage or the spy_touch_search, depending on the case.

Storage:

 ...
    spy_touch.id_touch AS %s,
    spy_touch.item_id AS %s,
    spy_touch_storage.id_touch_storage AS %s,
 ...
   INNER JOIN spy_touch t ON (tree.id_category_node = t.item_id AND t.item_type = :itemType)
   LEFT JOIN spy_touch_storage ON spy_touch_storage.fk_touch = t.id_touch AND spy_touch_storage.fk_locale = :fk_locale

Search:

 ...
    spy_touch.id_touch AS %s,
    spy_touch.item_id AS %s,
    spy_touch_search.id_touch_search AS %s,
 ...
   INNER JOIN spy_touch t ON (tree.id_category_node = t.item_id AND t.item_type = :itemType)
   LEFT JOIN spy_touch_search ON spy_touch_search.fk_touch = t.id_touch AND spy_touch_search.fk_locale = :fk_locale

:itemType is the collectorType and :fk_locale is idLocale from spy_locale table. Each placeholder (%s) for the Touch columns will be replaced by values that are defined in the CollectorConfig. Check prepareCollectorScope() methods for implementation details.

Propel Touch Record Implementation

For Propel type collector query those steps are not required, as the information about Touch Record is already integrated.

Configure

The collectors register through plugins. When adding a new collector, you must create a dedicated plugin for it in the Communication layer, under the Plugin/ folder. The plugin for the new collector must extend AbstractCollectorPlugin class and must implement the run() method which calls the corresponding CollectorFacade that calls the associated collector.

<?php
/**
 * @param \Orm\Zed\Touch\Persistence\SpyTouchQuery $baseQuery
 * @param \Generated\Shared\Transfer\LocaleTransfer $locale
 * @param \Spryker\Zed\Collector\Business\Model\BatchResultInterface $result
 * @param \Spryker\Zed\Collector\Business\Exporter\Writer\WriterInterface $dataWriter
 * @param \Spryker\Zed\Collector\Business\Exporter\Writer\TouchUpdaterInterface $touchUpdater
 * @param \Symfony\Component\Console\Output\OutputInterface $output
 *
 * @return void
 */
abstract public function run(
    SpyTouchQuery $baseQuery,
    LocaleTransfer $locale,
    BatchResultInterface $result,
    WriterInterface $dataWriter,
    TouchUpdaterInterface $touchUpdater,
    OutputInterface $output
);
?>

Example: NavigationCollectorStoragePlugin.

<?php
namespace Pyz\Zed\Collector\Communication\Plugin;

use Generated\Shared\Transfer\LocaleTransfer;
use Orm\Zed\Touch\Persistence\SpyTouchQuery;
use Spryker\Zed\Collector\Business\Exporter\Writer\TouchUpdaterInterface;
use Spryker\Zed\Collector\Business\Exporter\Writer\WriterInterface;
use Spryker\Zed\Collector\Business\Model\BatchResultInterface;
use Spryker\Zed\Collector\Communication\Plugin\AbstractCollectorPlugin;
use Symfony\Component\Console\Output\OutputInterface;

/**
 * @method \Pyz\Zed\Collector\Communication\CollectorCommunicationFactory getFactory()
 * @method \Pyz\Zed\Collector\Business\CollectorFacade getFacade()
 */
class NavigationCollectorStoragePlugin extends AbstractCollectorPlugin
{

    /**
     * @param \Orm\Zed\Touch\Persistence\SpyTouchQuery $baseQuery
     * @param \Generated\Shared\Transfer\LocaleTransfer $locale
     * @param \Spryker\Zed\Collector\Business\Model\BatchResultInterface $result
     * @param \Spryker\Zed\Collector\Business\Exporter\Writer\WriterInterface $dataWriter
     * @param \Spryker\Zed\Collector\Business\Exporter\Writer\TouchUpdaterInterface $touchUpdater
     * @param \Symfony\Component\Console\Output\OutputInterface $output
     *
     * @return void
     */
    public function run(
        SpyTouchQuery $baseQuery,
        LocaleTransfer $locale,
        BatchResultInterface $result,
        WriterInterface $dataWriter,
        TouchUpdaterInterface $touchUpdater,
        OutputInterface $output
    ) {
        $this->getFacade()
            ->runStorageNavigationCollector($baseQuery, $locale, $result, $dataWriter, $touchUpdater, $output);
    }

}
?>

Of course, for each collector a dedicated method must be defined in the CollectorFacade, that makes the connection between the collector plugin and the collector.

<?php
/**
 * @param \Orm\Zed\Touch\Persistence\SpyTouchQuery $baseQuery
 * @param \Generated\Shared\Transfer\LocaleTransfer $locale
 * @param \Spryker\Zed\Collector\Business\Model\BatchResultInterface $result
 * @param \Spryker\Zed\Collector\Business\Exporter\Writer\WriterInterface $dataWriter
 * @param \Spryker\Zed\Collector\Business\Exporter\Writer\TouchUpdaterInterface $touchUpdater
 *
 * @return void
 */
public function runStorageNavigationCollector(
    SpyTouchQuery $baseQuery,
    LocaleTransfer $locale,
    BatchResultInterface $result,
    WriterInterface $dataWriter,
    TouchUpdaterInterface $touchUpdater,
    OutputInterface $output
) {
    $this->getFactory()
        ->createStorageNavigationCollector()
        ->run($baseQuery, $locale, $result, $dataWriter, $touchUpdater, $output);
}
?>

The collectors are registered in the application through their corresponding plugins in the CollectorDependencyProvider class.

the collectors that gather data for search type store are registered under SEARCH_PLUGINS

the collectors that gather data for the storage type store are registered under STORAGE_PLUGINS

<?php
$container[static::SEARCH_PLUGINS] = function (Container $container) {
    return [
        'product_abstract' => new ProductCollectorSearchPlugin(),
    ];
};

$container[static::STORAGE_PLUGINS] = function (Container $container) {
    return [
        'product_abstract' => new ProductCollectorStoragePlugin(),
        'categorynode' => new CategoryNodeCollectorStoragePlugin(),
        'navigation' => new NavigationCollectorStoragePlugin(),
        'translation' => new TranslationCollectorStoragePlugin(),
        'page' => new PageCollectorStoragePlugin(),
        'block' => new BlockCollectorStoragePlugin(),
        'redirect' => new RedirectCollectorStoragePlugin(),
        'url' => new UrlCollectorStoragePlugin(),
    ];
};
?>

Collect

Data collection is done in 2 steps:

Querying/fetching data from the SQL database (Persistence layer)

Aggregating/processing data (Business layer)

Query

To fetch data from the database, you can either use a Propel or PDO type query, under the Persistence layer.

  • AbstractPdoCollectorQuery - uses native SQL
  • AbstractPropelCollectorQuery - uses Propel Query

AbstractPdoCollectorQuery

AbstractCollectorQuery contains one abstract method prepareQuery().

<?php
/**
 * @return void
 */
abstract protected function prepareQuery();
?>

You can set your own SQL query with sql() method of CriteriaBuilder interface.

<?php
/**
 * @param string $sqlTemplate
 *
 * @return $this
 */
public function sql($sqlTemplate);
?>

Example of the NavigationCollector query, that uses native PostgreSQL to retrieve hierarchical navigation data.

CriteriaBuilder

The collector will use conditions generated by CriteriaBuilder to create data set for synchronization.

CriteriaBuilder provides an interface that that makes sure all the parameters and their values are properly bind, or if the order and limit a properly setup, and if they are in the right place in SQL query.

AbstractPropelCollectorQuery

Implementation of AbstractCollectorQuery is fairly simple, as it has only one abstract method prepareQuery().

<?php
/**
 * @return void
 */
abstract protected function prepareQuery();
?>

You define your criteria with touchQuery interface, using Propel Query.

Example of BlockCollector query, which uses Propel Query to retrieve CMS blocks data.

Process

There are four collector types, grouped by store and query type they use.

Storage Collectors

  • AbstractStoragePdoCollector - uses PDO type query
  • AbstractStoragePropelCollector - uses Propel type query

The entities processed by those collectors will be synchronized with Storage type data store.

Search Collectors

  • AbstractSearchPdoCollector - uses PDO type query
  • AbstractSearchPropelCollector - uses Propel type query

The entities processed by those collectors will be synchronized with Search type data store.

After fetching data from the Persistence layer, and processing in Buissiness layer, it is saved under the TouchKey in the data store. A TouchKey is any unique string that can be used to identify resources in the stores.

There are two abstract methods, which every collector has to implement.

<?php
/**
 * @param string $touchKey
 * @param array $collectItemData
 *
 * @return array
 */
abstract protected function collectItem($touchKey, array $collectItemData);

/**
 * @return string
 */
abstract protected function collectResourceType();
?>

The string returned by collectResourceType() is used to generate the TouchKey.

The collectItem() method decides which data is saved under the TouchKey in the store.

Example of a RedirectCollector class, which uses Storage type data store.

<?php
namespace Pyz\Zed\Collector\Business\Storage;

use Spryker\Zed\Collector\Business\Collector\Storage\AbstractStoragePropelCollector;
use Spryker\Zed\Url\UrlConfig;

class RedirectCollector extends AbstractStoragePropelCollector
{

    /**
     * @return string
     */
    protected function collectResourceType()
    {
        return 'redirect';
    }

    /**
     * @param string $touchKey
     * @param array $collectItemData
     *
     * @return array
     */
    protected function collectItem($touchKey, array $collectItemData)
    {
        return [
            'from_url' => $collectItemData['from_url'],
            'to_url' => $collectItemData['to_url'],
            'status' => $collectItemData['status'],
            'id' => $collectItemData['id'],
        ];
    }

}
?>

The output of this collector will produce a redirect type entity, stored under kv:de.de_de.resource.redirect.1 key, with the following content:

{
    "from_url": "/imp",
    "to_url": "/impressum",
    "status": 303,
    "id": 1
}

The content of $collectItemData array represents one row from the result set, generated by the collector’s query in Persistence layer.

Related Topics Link IconRelated Topics