Mercari Dataflow Template (MDT) is an OSS tool for easy data processing using GCP’s distributed data processing service, Cloud Dataflow.
It is used within Merpay, Inc. to combine, process, and store data between various data sources.
In this article, I will introduce examples of using MDT to input/output and process data to/from Cloud Spanner.

How to use Mercari Dataflow Template ?

First, you need to deploy MDT. After that, describe the configuration file (called a pipeline file) that defines the process you want to execute in JSON format. Upload that file to GCS, and launch it using the gcloud command or Dataflow’s REST API.
In this section…


Mercari Dataflow Template (MDT) is an OSS tool for easy data processing using GCP’s distributed data processing service, Cloud Dataflow.
It is used within Merpay, Inc. to combine, process, and store data between various data sources.
In this article, I will introduce examples of using MDT to input/output and process data to/from Cloud Spanner.

How to use Mercari Dataflow Template ?

First, you need to deploy MDT. After that, describe the configuration file (called a pipeline file) that defines the process you want to execute in JSON format. Upload that file to GCS, and launch it using the gcloud command or Dataflow’s REST API.
In this section…

Yoichi Nagai

Data Engineer at Merpay, Inc.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store