The Spring Cloud Data Flow team is pleased to announce the release of 2.0 RC1 of Data Flow. Follow the Getting Started guides for running on Local, Cloud Foundry, and Kubernetes.

Hand in hand is the 2.0 RC1 release of Spring Cloud Skipper. The getting started section in the reference guide is the best place to start if you want to use Skipper separately from Data Flow.

Here are the highlights for Data Flow

• Revamped metrics and monitoring of deployed applications on Kubernetes

• Updated analytics using micrometer

• Security improvements

• Dashboard improvements

• Database migration support

Revamped metrics and monitoring of deployed applications on Kubernetes

Data Flow 2.0 introduces a new architecture for collecting and displaying application metrics for Streams. The Data Flow Metrics Collector introduced in Data Flow 1.4 has been removed.

This new architecture is based on using the Micrometer library in the deployed applications to send metrics to popular monitoring systems and then visualizing the metrics with Grafana. Each application in the Einstein release train of the application starters contains Micrometer libraries for Prometheus, InfluxDB and DataDog.

In this release we provide instructions on how to get started using Prometheus as the monitoring system when running Data Flow on Kubernetes. Two Grafana dashboards are also provided, one that shows a more application centric view and other that show a more stream based view. Here is a screenshot of the Grafana dashboard based on Prometheus showing the message rates for the applications in a stream.

Selectors for applications and streams allows you to navigate down to specific areas of interest.

Updated analytics using micrometer

In Data Flow 2.0, the use of Redis counters has been replaced with counters provided by the Micrometer library. The Einstein release train of application starters now provide a Counter Processor and a Counter Sink. The twitter analytics sample application has been updated to use the new counter implementation and a Grafana Dashboard is provided to show the equivalent view of what the Data Flow Analytics UI had previously shown.

Security improvements

The server uses OAuth 2.0 authentication to secure the REST endpoints. Documentation and sample code is now available that shows how to configure the DataFlowTemplate for use with OAuth2.0. Dependent libraries that had recent CVEs reported were updated.

Dashboard improvements

You can now search the audit trail by a date range.

Database migration support

Flyway migration code has been improved and tested across all supported databases using a suite of Docker based integration tests.

What is next

We postponed the feature of storing the detailed history of Task deployments for 2.1. Various documentation issues and additional testing is planned leading up to the 2.0 GA release. General themes for 2.1 will be a focus on documentation, getting started guides, videos and overall user experience. A new site for Data Flow is planned as well.

Also note that the 1.x line of Spring Cloud Data Flow will cease maintenance twelve months from the 2.0 GA announcement date. The 2.0 GA is tentatively planned for end of February 2019, so in February 2020, the 1.x line would reach end-of-general-support status.

A special thanks

Last but not least, we wanted to thank Daniel Serleg for his many contributions in this release.

Stay in touch…​

As always, we welcome feedback and contributions, so please reach out to us on Stackoverflow or GitHub or via Gitter.