Apache Beam Reviews

Beam is awesome. Both Batch and Stream processing handled via the same semantics/code. Just switch out your IOs, ex: database or files for batch, or a message queue for streaming use cases

People sometimes have trouble learning. I wish it was easier for people

We use Beam to handle most of our data processing needs. Both Batch and Stream use cases, and across our clouds and with a bunch of inputs and outputs.

I have used Apache Beam to design my pipeline on Google cloud platform. My experience was seamless. I have designed for both batch and streaming data.

Lack of available courses on apache beam which can help newbies to learn Beam

I could easily recommend anyone to use Apache Beam for its simplicity and detailed documentation.

I have used windowing technique to accommodate late dat within my pipeline which was a brilliant task.

Liked the way Beam provides abstraction for the complex distributed paradigm using pCollection and Transforms etc.

Also its unified approach to deal with both Batch and Stream processing is unique and efficient.

At the time of doing development , i found doing Join operation was not that simple and we had to use coGroupByKey for that which was a bit confusing for us. May be providing abstraction as simple join operation would help the users community. May be the developer community might have already provided that option but it was not there when we were doing our use case.

We have been using Beam for a batch data processing with Source as Google Cloud Storage and Destination is Big Query.

Frequently update and supports multiple language

More tedious in debugging the application code. Also, not very informative error logs.

Do more research before you get started to see if this is a suitable product for you.

Creating a pipeline for collecting data from IOT devices to analyse the insights provided by them. The data are health care data and navigation data.

In terms of capabilities and generality, I think Apache Beam is currently the most advanced and most flexible framework for designing and implementing modern data-intensive applications. Perfectly able to specify both batch and streaming computations, plus in terms of streaming capabilities, it offers really a lot! It takes some time to get comfortable with the APIs, but after that is goes pretty smoothly. Finally the support for various execution engine thanks to the "runner" abstraction makes it a great choice when portability across different platforms either on premise or in the Cloud becomes a factor. Definitely recommended.

Currently the APIs are not exactly super friendly.

Definitely recommended for modern data pipeline.

Implementing scalable ETL pipelines running on the Cloud. Implementing business triggers based on a high workload of incoming events.

With Apache beam, data processing pipelines are made really easy. We've been using this over the past 3 months and found it really successful especially for the ETL process of data we produce.

It is good for smaller processes, but not advisable for large streamline processes.

Just use it, it is worth it.

We do ETL and producing features in ML.

It's an open-source workflow engine that lets you code your logic of data processing workflow. It allows you to process your data based on batch and streams.

You should learn the basic function of your workflow using Java. So, it takes time (depends on your learning curve)

it's a great streaming data processing workflow engine that you could customize your own logic to your workflow engine.

streaming data from

The SDKs are good and easy to use. It's amazing how fast you get something up and running.

Not yet support for python3 and still there are some benefits to using JAVA and not really a fan of JAVA

Just jump right into it.

Supported by google. Is a solution that has everything, many systems can handle both batch and streaming, but they often do so via separate APIs.

Beam focus on capturing properties of your data and your logic, instead of letting details.

The project uses the correct programming model for streaming and batch data processing.

capabilities will not always be exactly the same across different Beam runners at a given point in time

.fast implementation of all methods to try change data

Difficult to use and ehat exactly you need to do

Clean data and save it on bigquery (gogle cloud)