Real-time complex data analytics and machine learning by using Apache Spark and StreamSets Transformer

In today's world data is growing at exponential rate with more than 90% of all data being generated in the last two years in different formats.In reality, many companies are still stuck with an old architectures styles and technologies, relying on once per-day ETL (Extract Transformation Load) job to load daily collected data from transactional relational databases into Data Warehouse or Hadoop in the best case.But what if you want to process data from combination of on-prem and Cloud data sources in a real time, with complex data transformations on any data types - not just relational, and eventually add AI/ML (Machine Learning) in data pipeline as a bonus?Maybe you are struggling with an exponential growth of collected data, and your nightly ETL batches cannot complete on time because it has to run in parallel with busy relational database backup jobs?You may also not be happy with your data warehouse reporting system because it does not display actual data due to ETL DWH job which runs once per day.Or you might want to deliver your data not only to your local on-prem system, but also in one or more Public Cloud at the same time for further processing?You may also experience performance issues because your Data Warehouse just cannot scale out to deal with an exponential data growth.At the same time, you may wish to have real-time end-to-end monitoring and alerting of your data pipelines equipped with all performance reports to further tune your data processing?If solving all of those challenges sounds like a dream, you might be interested to come and see possible solutions based on DataOps and modern Big Data and AI/ML technologies.