Big Data

Python and Its Use for Data Science Python is easy to learn, and its syntax is relatively simple. It is a popular language for data science because it is powerful and easy to use. Python is an excellent language for data analysis because it includes a variety of data structures, modules, and tools. There are many reasons why you should use Python for data science: Python is a very versatile language. It can be used for a wide variety of data science tasks, from data preprocessing to machine learning and data visualization. Python is very easy to learn. You don't need to be an expert in computer science to start using Python for data science. In fact, most data science tasks can be done with just a few simple Python commands. Python is supported by a wide range of libraries and tools. This means you can easily find the tools and libraries you need to carry out your data science tasks. Some Key Data Science Libraries in Python There are a few python libraries with data science capabilities that are worth mentioning. NumPy is a popular library for data analysis and scientific computing. It has a wide range of data structures, including arrays, lists, tuples, and matrices. IPython is an interactive shell for Python that makes it easy to explore data, run code, and share results with other users. It provides a rich set of features for data analysis, including inline plotting and code execution. SciPy is a collection of mathematical libraries for data analysis, modeling, and scientific computing. It includes tools for data handling, linear algebra, imaging, probability, and more. Pandas is a powerful library for data analysis and data visualization. It has a few unique features, including data frames, which are similar to Excel sheets but can hold a lot more data, and powerful data analysis operations, such as sorting and grouping. Improving Data Science Work With Python There are many ways to improve data science work with Python. Here are a few tips: Use a data science library. Many data science libraries, such as pandas, scikit-learn, and numpy, provide convenience functions for common data analysis tasks. Use a data visualization library. Many data visualization libraries, such as matplotlib and ggplot2, provide convenient functions for creating graphs and charts. Use a c. data preprocessing libraries, such as pandas’ dataframe.to_csv() and scikit-learn’s sklearn. There are many ways to preprocess data for machine learning, but two of the most popular are pandas' dataframetocsv and scikit-learn's sklrearn. preprocessing. Advanced Python for Data Science Topics First, I will discuss how to use pandas. Pandas is a data analysis library that makes it easy to work with data frames, data sets, and data analysis operations. It offers a high-level interface to data, making it easy to access and work with data. Pandas can handle data of various types, including NumPy arrays, text files, and relational databases. Pandas also have powerful data analysis tools, including data plotting and data analysis functions. Pandas can help you analyze your data quickly and easily. Second, I will be discussing how to use NumPy. NumPy is a powerful Python library that makes working with large, multi-dimensional arrays and matrices much easier. NumPy also provides a host of other useful features, such as tools for integrating C/C++ code, linear algebra routines, and Fourier transform capabilities. If you're doing any kind of scientific or numerical computing in Python, NumPy is worth checking out. One of the most important features of NumPy is its ability to perform vectorization. Vectorization is a powerful technique that can greatly improve the performance of your code. NumPy provides an easy-to-use interface for vectorizing your code. Simply add the @vectorize decorator to any function that you want to vectorize. Last, I will be discussing how to use SciPy. SciPy is a Python-based ecosystem of open-source software for mathematics, science, and engineering. It includes modules for linear algebra, optimization, integration, interpolation, special functions, FFT, signal and image processing, ODE solvers, and more. The SciPy library is built to work with NumPy arrays and provides many user-friendly and efficient numerical routines, such as routines for numerical integration and optimization. In addition, SciPy provides a large number of high-level scientific functions such as statistical tests, root-finding, linear algebra, Fourier transforms, and more. SciPy is an active open-source project with an international team of developers. It is released under the BSD license and is available for free. Data Science Projects You Can Try With Python Here are some examples of Python data science projects that you can try: 1. Predicting the Stock Market: You can use Python to predict the stock market. This is a great project for beginners because it doesn’t require a lot of data. 2. Analyzing the Enron Email Dataset: The Enron email dataset is a great dataset for data science projects. You can use Python to analyze emails and find out interesting insights. 3. Classifying Images with a Convolutional Neural Network: You can use a convolutional neural network to classify images. This is a great project for people who are interested in machine learning. 4. Analyzing the Yelp Reviews Dataset: The Yelp reviews dataset is a great dataset for data science projects. You can use Python to analyze the reviews and find out interesting insights. 5. Predicting House Prices. As a real estate agent, one of the most important skills is predicting house prices. This can be difficult, as many factors go into pricing a home. However, with the right data and a bit of Python programming, it is possible to create a model that can accurately predict home prices. The first step is to collect data on recent home sales in your area. This data should include the sale price, square footage, number of bedrooms and bathrooms, and any other relevant information. You can either find this data online or collect it yourself from public records. Once you have this data, you will need to clean it up and prepare it for use in a machine-learning model. This includes removing any missing values and ensuring that all the data is in the correct format. Next, you will need to choose a machine learning algorithm that you will use to train your model. Python is not only one of the most popular programming languages but also one of the most beautiful languages to look at. While many languages use punctuation and keywords that can look like gibberish to the untrained eye, Python's syntax is clean and elegant. Even beginners can quickly learn to read and write Python code. And it's not just the syntax that makes Python beautiful. The language also has a philosophy known as Python Zen, which encourages developers to write code that is simple, readable, and maintainable. This philosophy has helped to make Python one of the most popular languages for beginners and experienced developers alike.

Data science and software engineering are IT-based domains and play widespread organizational functions. Both domains require a wide range of programming skills from different domains. The career opportunities in these fields are increasing day by day. The report, titled "Analytics & Data Science Jobs in India 2022," showcases the following results: Compared to the 9.4% global open jobs in June 2021, 11.6% of the total open positions came from India alone. An increase in open positions observed by captive centers, ongoing domestic company investments in establishing analytics and data science capabilities, a significant shift of domestic and foreign IT and KPO organizations to India, increased funding for AI and analytics-based start-ups in India, and an increase in open positions are all factors contributing to this growth. With 51,149 available positions or 28.5% of the market, Bengaluru overtook Delhi NCR as the top location for analytics and data science experts for the sixth consecutive year. Employees are returning to their jobs at a higher rate than in other cities, which supports the rise. Likewise, the demand for software engineering jobs is rising. Moreover, with the IoT and blockchain tech boom, software engineering is expected to retain its crown as one of the most sought-after degrees in India. Data Science Data science compromises the collection and analysis of data for logical and valuable outputs. It deals with an extensive range of data analytics using various tools and techniques such as AI, data mining, and machine learning. It includes studying various structured and unstructured data to derive expressive information and find unseen patterns to make effective business decisions. Data science is among one of the fastest-growing domains. It is an essential aspect of any organization as it makes decisions on a factual basis. Software Engineering Software engineering is a systematic method of creating software by engineering principles and comprises planning, building, designing, and testing software applications. It is a detailed study of development, design, and software maintenance, and it manipulates the best methods and techniques for creating high-quality software. It is a very prevalent career option for many candidates, providing thousands of opportunities in various national and multinational organizations. Qualifications for Data Science and Software Engineering Check out the qualifications needed to pursue data science and software engineering degrees individually. Data Science If a candidate wants to pursue their career as a data scientist, they should have completed their undergraduate and post-graduate from a well-recognized university. The degrees include: B. Tech or M. Tech majoring in computer science B.Sc or M.Sc in statistics/mathematics MBA Software Engineering If a candidate wants to pursue their career as a software engineer, they should know a few programming languages such as; Java, Javascript, Python, C, Ruby, and C#. They should have an undergraduate and postgraduate degree from a well-recognized university. The suitable domains include the following: Computer Programs Deep and Practical Programming Data Structures and Algorithms Data Scientist vs. Software Engineer There are a lot of differences between data science and software engineering. Here is a list of some of the main differences: Data Science Software Engineering A data scientist gathers data and mainly focuses on the processing of data. Software engineering develops applications and mainly focuses on features. Data science consists of statistics and machine learning. Software engineering prefers coding and mainly concentrates on it. Data science undertakes and deals with exploratory information. Software engineering is directed at building systems. Data science includes various skills. Some of them are machine learning, data visualization, and statistics. Software engineering also includes several skills, such as how to code and program in different languages. Data science deals with various tools such as Database tools, Data Visualization tools, and Data Analytics Tools. Software engineering also pacts with various instruments such as integration apparatus, CMS devices, programming instruments, testing devices, plan instruments, and database services. Data science is oriented to process. Software engineering is oriented to methodology. Data Science vs. Software Engineering Based on Career Opportunities If anyone is confused about whether to become a data scientist or a software engineer, here is a well-researched section about the career opportunities in data science and software engineering. One can consider this and choose a suitable domain for their career. A Career in Data Science The demand for data scientists is increasing promptly. Different opportunities are available according to the convenience of the candidates. One can become an analyst at an entry-level or pursue specialized courses to become a chief data officer. One can become an analyst in several domains: quantitative analyst, business analyst, marketing analyst, operations analyst, business intelligence analyst, data analyst, and systems analyst. If anyone wants to have senior roles, they can pursue some specialized courses in data science to become a senior data scientist, lead data scientist, or machine learning engineer. A Career in Software Engineering The field of software engineering is very broad, so it is beneficial to make a career in this field. After pursuing software engineering, one can become a full-stack engineer, mobile app developer, software development engineer, game engineer, front-end engineer, DevOps engineer, security engineer, data engineer, back-end engineer, embedded systems engineer, and graphics engineer. Suppose anyone wants to have a lead role. In that case, they can pursue a specialized course and become a technical architect, tech lead, chief technology officer, senior software engineer, team manager, or junior software engineer. Data science and software engineering are IT-based domains; they play widespread functions within an organization, requiring a wide range of programming skills from different domains. The career opportunities in these fields are increasing day by day. If anyone wants to pursue software engineering in data science, now is the right time to develop their careers. Data Science vs. Software Engineering Based on Salary Structure Data scientists and software engineers get a good amount annually, and these domains have benefits. Salary is usually based on performance, competence, and skills. There are some variations in the pay scale of both domains. Given below are some of the salary differences between data science and software engineering: Data Scientist Salary The salary of data scientists depends on their prior experience in a particular field; in addition, an increment in their pay rate occurs every year. Salaries are different for different designations; for an entry-level data scientist in India, the annual salary is around INR 4,50,000. For a mid-level data scientist, the yearly wage is approximately INR 10,00,000, and for an experienced data scientist, the pay rate is about INR 20,00,000. Software Engineer Salary The salary of software engineers in India is on the higher side, and they receive a decent amount yearly. The demand for software engineering is increasing daily, so their pay rate is also enhancing accordingly. There are different levels in software engineering, and each level's salary is different. An entry-level software engineer receives around INR 3,50,000, and a mid-level software engineer receives about INR 5,60,000. The annual pay scale for an experienced software engineer is about INR 14,00,000. Conclusion This article was about data science and software engineering qualifications required career paths and salary. We hope you find the information helpful in choosing a career.

Apache Pulsar and Apache Kafka share a similar data model around logs. This makes it possible to implement a Kafka-compatible protocol handler so that Kafka users can migrate existing Kafka applications and services to Pulsar without modifying the code. It also allows these applications to leverage various Pulsar features such as multi-tenancy, infinite event stream retention, and serverless event processing. This is where the open-source project Kafka-on-Pulsar (KoP) comes in. Why Did the Community Develop KoP? To begin, I think it is important to answer a key question: “Why did you develop it?” As a maintainer of the KoP project, I notice that this is usually the first thing that comes to mind for a new community member. To answer this question, let’s have a quick look at the basics of Pulsar and Kafka. Pulsar and Kafka have some common concepts. For example, they both have producers and consumers with brokers hosting different topics to serve them. Additionally, both of them support topic partitions to achieve better parallelism. In Kafka, the client sends messages to the leader broker through the PRODUCE request. These messages are persisted to the node locally. Each follower reads the data from the leader through the FETCH request to store a copy of the messages. In this leader-follower architecture, each broker needs to handle both data processing and storage. One disadvantage of the design is that it guarantees the most recent and relevant data replica is only stored on the leader broker, which serves both producers and consumers. This means a Kafka cluster can be overwhelmed during traffic bursts as the load may not be spread across followers. By contrast, Pulsar separates serving (brokers) and storage (bookies) into different layers. At the computing layer, all Pulsar brokers are stateless and equivalent to each other. As shown in the figure below, the client sends messages to the broker through the SEND request. After processing the messages, the broker delivers them to different bookies through the ADD_ENTRY request. Specifically, data are written to bookies based on the configured writing strategy (that is, the values of ensemble size, write quorum, and ack quorum). This helps achieve data high availability across storage nodes. Most importantly, both brokers and bookies can be easily scaled at their own layer, not impacting the other. This cloud-native architecture of Pulsar features great scalability, availability, and resiliency, providing users with solutions to some key pain points in Kafka. Therefore, many users are looking for a graceful plan to migrate from Kafka to Pulsar. Migration Plans Before KoP was developed, migrating from Kafka to Pulsar was no easy task. I have noticed that some community members used and perhaps are still using the following ways: Update Clients Users need to rewrite their code and optimize some client-side configurations. Admittedly, this is not an ideal solution as it may incur additional costs. Apart from that, Pulsar’s ecosystem still has a long way to go compared with Kafka’s. The latter enjoys a more mature ecosystem in many ways, such as multi-language client support and third-party integrations. This means rewriting the code can cause many problems when you try to use tools in Kafka’s ecosystem. Pulsar Adaptor for Apache Kafka Initially, the Pulsar community tried to solve the migration issue by developing a tool called Pulsar Adaptor for Apache Kafka. It allows users to replace the Kafka client dependency with the Pulsar Kafka wrapper. It does not require any changes to the existing code. Nevertheless, its disadvantages are obvious: Only applicable to Java-based clients Problems in handling Kafka offsets Users still need to learn some Pulsar client configurations Kafka-on-Pulsar (KoP) To provide a smoother migration experience for users, the KoP community came up with a new solution. They decided to bring the native Kafka protocol support to Pulsar by introducing a Kafka protocol handler on Pulsar brokers. Protocol handlers were a new feature introduced in Pulsar 2.5.0. They allow Pulsar brokers to support other messaging protocols, including Kafka, AMQP, and MQTT. Compared with the above-mentioned migration plans, KoP features the following key benefits: No Code Change: Users do not need to modify any code in their Kafka applications, including clients written in different languages, the applications themselves, and third-party components Great Compatibility: KoP is compatible with the majority of tools in the Kafka ecosystem. It currently supports Kafka 0.9+ Direct Interaction With Pulsar Brokers: Before KoP was designed, some users tried to make the Pulsar client serve the request sent by the Kafka client by creating a proxy layer in the middle. This might impact performance as it entailed additional routing requests. By comparison, KoP allows clients to directly communicate with Pulsar brokers without compromising performance To learn more about the reasons for developing KoP, refer to the KoP white paper. How KoP Works KoP is implemented as a protocol handler plugin with the protocol name "Kafka." It is loaded when the Pulsar broker restarts. By default, Kafka and Pulsar clients can work at the same time. If you check the source code of Pulsar, you can find a class called ServerCnx, which exposes port 6650. After the Pulsar client sends requests to the port, ServerCnx then parses these requests so the broker can take action accordingly, such as reading and writing data from and to BookKeeper. After you enable KoP for your cluster, it exposes port 9092 so the Kafka client can send requests to it. Similar to ServerCnx, KoP processes these requests and then asks the broker to respond. Both ServerCnx and KoP have access to all broker resources, such as topics and subscriptions. Implementation and Design Let’s examine some key concepts, especially how they are implemented in KoP. Topics and Partitions In Kafka, all the topics are stored in one flat namespace. In Pulsar’s multi-tenant architecture, messages are organized in a three-level hierarchy of tenants, namespaces, and topics. A Pulsar topic can be either persistent or non-persistent. As Kafka persists, all messages and all topics in KoP are persistent. The following is the Pulsar topic naming convention (the partition is part of the topic name as the suffix): {persistent | non-persistent}://tenant/namespace/topic-partition-0 To enable Kafka users to leverage the multi-tenancy feature of Pulsar, KoP allows them to use a short topic name directly as it provides a default tenant and namespace. At the same time, users can also specify the Pulsar tenant and namespace so that clients can access other topics. See the following table for details: Kafka topic name Tenant Namespace Short topic name my-topic <kafkaTenant> <kafkaNamespace> my-topic my-tenant/my-ns/my-topic my-tenant my-ns my-topic persistent://my-tenant/my-ns/my-topic my-tenant my-ns my-topic In this table, kafkaTenant and kafkaNamespace define the default names that KoP uses for a short topic name, which default to public and default respectively. Authentication In production, you may have different namespaces for Pulsar clients and Kafka clients. To improve security, you can enable authentication for KoP. It uses Kafka SASL mechanisms and the token-based Authentication Provider in Pulsar for authentication. Therefore, you need to enable authentication for the following components: Pulsar Brokers KoP (You need to specify the mechanism by configuring saslAllowedMechanisms, which currently only supports PLAIN and OAUTHBEARER) Kafka Clients See the figure below for how the authentication works: A Kafka client sends the SASL_HANDSHAKE request with the mechanism configured for the Kafka client. KoP then verifies whether the mechanism is supported. If it is, KoP creates the SaslServer based on the mechanism. After KoP responds to the Kafka client, it sends the SASL_AUTHENTICATE request with a token back to KoP. KoP validates the token according to the SaslServer created before. For detailed information about how to configure authentication, see Security in Kafka on Pulsar. Authorization You can also configure authorization in KoP to define the permissions that different clients have. For example, some clients can only produce messages while others may be able to consume messages. Kafka uses ACLs (Access Control Lists) and resource types to control operations available for different resources. KoP supports the Topic resource type. See the figure below to understand how it maps different ACL operations: Request ACL Resource Type Pulsar permissions FETCH, OFFSET_COMMIT READ TOPIC canConsume PRODUCE WRITE TOPIC canProduce METADATA, LIST_OFFSET, OFFSET_FETCH DESCRIBE TOPIC canLookup CREATE_TOPICS, DELETE_TOPICS, CREATE_PARTITIONS CREATE, DELETE, ALTER TOPIC allowNamespaceOperation After authentication is finished, KoP obtains an authorization ID (or a Pulsar role) and sends it to AuthorizationService on the broker. Note: A Pulsar role is a string, which can represent a single or multiple clients. Roles are used to manage client permissions, such as producing or consuming messages on certain topics. Pulsar uses authentication providers to establish the identity of a client and then assign a role token to that client. This role token is then used for authorization and ACLs to determine what the client is authorized to do. Group Coordinator Implementing the Kafka group coordinator for KoP is very challenging. This is because Pulsar does not have a centralized group coordinator for assigning partitions to consumers or managing offsets. The group coordinator has two key responsibilities: Rebalance: Assign different partitions to consumers in the same group (subscription) Commit offsets: Persist on the latest offset information on a special offset topic In KoP, the group coordinator logic is rewritten in Java, while it is slightly different from the original implementation in Kafka. KoP introduces a new component called Namespace Bundle Listener. In Kafka, follower nodes notify the leader of ISR change. This is different from Pulsar since Pulsar brokers cannot communicate with each other. Pulsar uses bundles as a sharding mechanism. Topics are assigned to a particular bundle through the hash of the topic name. Note: This post does not explain bundles in detail. Here, you only need to know that the component monitors the change of bundles so the group coordinator can then assign partitions. Reads and Writes For the offset topic, the Pulsar producer and consumer perform the reads and writes directly. For ordinary messages, KoP uses the managed ledger and the managed cursor on the broker for reads and writes. Managed ledgers provide an additional storage abstraction on top of BookKeeper. A managed ledger manages several Pulsar segments (also known as ledgers in BookKeeper, which are used to store messages on bookies). It contains their size and state information. A managed cursor is a persisted cursor inside a managed ledger. It reads from the managed ledger and signals when the consumer finishes consuming messages. Keep in mind: KoP uses broker resources to directly read and write messages from and to BookKeeper instead of creating a client. This also reduces an additional network hop. Offset Implementation Kafka maintains a numerical offset for records in a partition, which indicates the position of the consumer in the partition. In Pulsar, there is no equivalent concept to the Kafka offset (Pulsar uses cursors to track the consumption and acknowledgment information of subscriptions. Note that cursors work in a more sophisticated way than Kafka offsets). Early KoP versions handled offsets with a simple conversion method, while it did not allow continuous offsets and could easily lead to problems. To solve this issue, the KoP community introduced a new concept “broker entry metadata” in KoP 2.8.0 to enable continuous offsets. As this is a very complicated topic, I refer you to one of my previous blogs Offset Implementation in Kafka-on-Pulsar to have a comprehensive understanding of offset implementation in KoP. Get Started With KoP If you have an Apache Pulsar cluster, you can enable KoP on your existing Pulsar cluster by downloading and installing the KoP protocol handler on Pulsar brokers directly. To get started, perform the following steps: Download the nar file on the Releases page, and move it to the protocols folder (create it manually) in the directory that stores your Pulsar package. Alternatively, clone the KoP GitHub repository and build the project locally In broker.conf or standalone.conf, configure the following fields: messagingProtocols=kafka allowAutoTopicCreationType=partitioned listeners=PLAINTEXT://127.0.0.1:9092 brokerEntryMetadataInterceptors=\ org.apache.pulsar.common.intercept.AppendlndexMetadatalntercepor Start your Pulsar brokers. When choosing different versions of KoP, keep in mind that: A KoP version has four minor numbers, namely x.y.z.m. The first three numbers refer to the Pulsar version that this KoP version is compatible with (for example, KoP 2.9.1.2 is compatible with Pulsar 2.9.1) When a minor Pulsar version is upgraded, the corresponding old version of KoP will not be maintained. For example, after Pulsar 2.9.2 was released, the KoP community did not maintain KoP 2.9.1.m Pulsar versions earlier than 2.8 are not recommended When you build the KoP project from the source code, do not use the master branch. Use a branch that is specific to the Pulsar version you are using FAQs About KoP I have collected some questions from the KoP community and listed my solutions and suggestions here. How Do I Specify entryFormat? entryFormat defines the format of KoP entries. Allowed values are pulsar, kafka, and mixed_kafka. It defaults to pulsar, which means Kafka clients can consume messages produced by Pulsar clients and that Pulsar clients can also consume messages produced by Kafka clients. However, the message decoding and encoding process can be time-consuming. Kafka-formatted messages need to be decompressed to recreate Pulsar-formatted messages Pulsar-formatted messages need to be decompressed to recreate Kafka-formatted messages For most applications in production, I suggest you configure entryFormat=kafka. In this way, KoP adds a key value to the metadata of a message to indicate that it comes from Kafka clients and then writes it into BookKeeper without unnecessary encoding and decoding. The downside of this method is that Pulsar clients cannot consume messages produced by Kafka clients. As these messages are written to bookies in the original format of Kafka, Pulsar clients cannot recognize them. To address this issue, the community designed a message payload processor for the Pulsar client in KoP 2.9.0. It allows Pulsar consumers to consume messages produced by Kafka producers even if you set entryFormat to kafka. You can configure the processor in your consumer application using messagePayloadProcessor as follows: Java final Consumer<byte[]> consumer = client.newConsumer() .topic("my-topic") .subscriptionName("my-sub") .messagePayloadProcessor(new KafkaPayloadProcessor()) .subscribe(); To import KafkaPayloadProcessor, you need to add the following dependency. Note that pulsar.version should be the same as the version of your pulsar-client dependency. <dependency> <groupId>io.streamnative.pulsar.handlers</groupId> <artifactId>kafka-payload-processor</artifactId> <version>${pulsar.version}</version> </dependency> For more information, see the KoP documentation. Consumers Couldn’t Keep Up With the Rate of Message Production A simple solution to this problem is increasing the value of maxreadEntriesNum: maxreadEntriesNum=5 This property represents the maximum number of entries that are read when KoP processes a FETCH request each time. Increasing this value reduces the number of requests required over the network. However, if it gets too large, it may put additional pressure on the memory. Note that the ManagedCursor currently does not check the byte-based read limit. Why Couldn’t I See Consumption Statistics When Using Pulsar-Admin? The community removed topic consumption statistics in KoP 2.8.0 and later versions. If you use pulsar-admin to check the information about a topic, you can’t see any available consumption details. The reasons for the removal are: Originally, when KoP committed an offset, it needed to find the message ID of the offset so the message could be acknowledged. This process is too time-consuming Kafka manages offset through the group coordinator. However, The coordinator broker may not be the leader broker. In Pulsar, only the topic owner broker can acknowledge messages KoP now provides a variety of Prometheus-based metrics to display consumption information, which is accessible through Grafana. For more information, see KoP metrics. Error Message NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender) When Pulsar automatically deletes partitioned topics (by default, inactive topics are deleted automatically in Pulsar), it does not delete the partition number metadata, which is stored in ZooKeeper. As a result, when KoP receives a request about the metadata from a client, it can still find the topic. KoP then sends the leader broker information to the client, which leads to the following error. Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender) To solve this problem, use either of the following ways: Disable automatic deletion of inactive topics in broker.conf: brokerDeleteInactiveTopicsEnabled=false Enable automatic deletion of the metadata of inactive partitioned topics in broker.conf. This is the preferred solution if you don’t want to disable topic automatic deletion in production: brokerDeleteInactivePartitionedTopicMetadataEnabled=true What’s New in KoP? Support for KoP in the OpenMessaging Benchmark Framework First, let me briefly explain what the OpenMessaging Benchmark Framework is and the scripts and configurations you need if you want to use it. The OpenMessaging Benchmark Framework is a suite of tools that make it easy to benchmark distributed messaging systems in the cloud. OpenMessaging benchmarking suites are currently available for systems such as Apache Pulsar, Apache Kafka, Apache RocketMQ, Redis, and Pravega. It provides the following for each supported message system: A Terraform script that automatically applies for and creates resources on cloud providers (for example, AWS) An Ansible script that automatically installs all necessary components (MQ, monitoring, and benchmark services) on remote machines A Java-based driver that allows you to easily run the MQ client. You can customize necessary client configurations in a YAML file Previously, when you tested the performance of KoP using the OpenMessaging Benchmark Framework, you had to manually change the Pulsar Ansible script by adding KoP-related logic. This is because the script does not deploy the Kafka protocol handler by default. In addition, you must use the Kafka driver and configurations to start Kafka producers and consumers in the test. To simplify the performance test of KoP using the benchmark, the KoP community created a driver specifically for KoP. It supports Kafka and Pulsar clients. In other words, you can have a mixed combination of Kafka and Pulsar producers and consumers working at the same time. It contains the logic of the payload processor, which allows Pulsar clients to consume Kafka-formatted messages if you set entryFormat to kafka. This is an example of using the KoP driver: ./bin/benchmark -d driver-kop/kafka_to_pulsar.yaml -o kop.json workloads/1-topic-1-partition-100b.yaml ./bin/benchmark -d driver-kop/kafka_to_kafka.yaml -o kop.json workloads/1-topic-1-partition-100b.yaml ./bin/benchmark -d driver-kop/pulsar_to_kafka.yaml -o kop.json workloads/1-topic-1-partition-100b.yaml For more information, see the driver-kop directory on the OpenMessaging Benchmark Framework GitHub repository. Note: The configuration of protocol handlers has also been simplified. Users can easily customize the protocol_handlers variable by specifying the protocol type, configuration file, and download URL. See this pull request and Apache Pulsar benchmarks to know more details. Automatic Deletion of Inactive Group Information on Znodes KoP leverages ZooKeeper to store the group information of consumers when it handles the FETCH request from them. The group information stored on znodes can then be used to update consumer metrics. The reason why KoP uses ZooKeeper for this purpose is that you cannot obtain the group information directly in the FETCH request. As shown in the figure below, a consumer sends different requests to the leader broker and the coordinator broker, while these two brokers may not be the same. Previously, the group information was never deleted on znodes, which could become extremely large and cause other problems to the KoP cluster. To solve this problem, the community found a way to automatically delete the group information when the group is not active anymore. In addition to the above improvements, the KoP community has also made some progress in fixing existing bugs and adding some minor features. Take a look at some of them if you are interested. Bug fixes: PR 1038, PR 973, PR 1125, PR 1230, and PR 1276. New features: PR 1006 and PR 1125. What’s Next? The KoP community looks to improve the project in the following ways. Add the Schema Support: Confluent Schema Registry API compatibility Add the compatible serializer and deserializer so that Kafka clients can interact with the Pulsar schema Improve the transactions feature Do some research on the OAuth2 plugin for other languages, such as C, C++, Go, and Python Finish the performance test report of KoP If you take a step back and reflect on how far KoP has come, it is not surprising to find that there are an increasing number of users joining the KoP community and making contributions in different ways. Like other tools in the Apache Pulsar ecosystem, KoP could not have come this far without the efforts of every community member involved. If you are interested in KoP, feel free to submit a pull request or open an issue on its GitHub repository.

Humanity wonders about tomorrow and predicting industry trends or seasonal consumer demand matters to clever business leaders. Predictive data analytics helps managers and their teams develop strategies for successful sales. Simultaneously, data scientists uncover hidden insights in business data. This post will explore the difference between predictive analytics and data science. Predictive Analytics in Data Science You want to understand the context of data science consulting services and the scope of predictive analytics solutions to learn the relationship between their business models. While some of their activities often overlap, both techniques have unique processes and outcomes/deliverables. What Is Data Science? Data science is the art of identifying patterns in structured, unstructured, and semi-structured data types to enhance the effectiveness of corporate growth strategies. A data science consulting firm also prepares and cleans the datasets while implementing machine learning (ML) models. Different data structures are always more demanding on computer systems because they primarily provide numerical calculations for structured data. Therefore, data science professionals (data scientists) develop several abstract models and adjust them for practical business applications. Data science models can integrate predictive data analytics solutions when testing different hypotheses/thought experiments corresponding to business decisions. Machine learning enhancements allow data scientists to teach their developed business models by re-testing business problems.  What Is Predictive Analytics? Data analytics comprises objective investigation into structured datasets to acquire beneficial insights into how the business has been performing since its incorporation. So, data analysts deal with structured data to learn about the past and present dynamics of an organization’s business unit. However, you can extend the resulting trends or performance curves while expecting external variables to stay constant. Leveraging historical data models for financial and performative forecasts is the essence of predictive analytics solutions. The reliability of business predictions relies on the behavior of their performance curves and context. For example, in long-term planning, you want to consider the year-on-year growth rate in sales forecasts. However, predictive modeling for seasonal or festive sales will often require different data analytics solutions. Skills in Predictive Analytics and Data Science Data science jobs and careers that involve developing predictive models require technological, mathematical, and social skills, which you can find below. Skillset for Predictive Analytics Roles Mathematics involves the geometric visualization of equations and functions, and a predictive model requires these abilities to ensure high-quality statistical modeling presentation. A predictive data analyst must also be familiar with computing languages and the commands needed to develop insights. Database management principles and philosophies allow professionals to improve data structure and integrity. Therefore, the analysts can save tremendous effort and streamline operations. The lack of the latest information technology (IT) skills poses a massive challenge in getting and keeping jobs in firms that provide predictive data models and corporate solutions. So, continuous professional development (CPD) in statistical science and coding languages is essential. Requirements in Data Science Consulting Jobs Data scientists require advanced statistical tools and techniques, without which they cannot discover hidden patterns in the datasets. Also, machine learning skills are indispensable to the data science job description. You want to develop your analytical and social skills to work with data engineers, analysts, and business management leaders. Likewise, you must know the core competencies in business administration to optimize the ML models for real insights. Example: Predictive Model Analytics for Data Science Consulting Solutions You might test the following hypothesis, which depicts a potential business decision affecting the marketing, customer care, and finance departments. “If we reduce the marketing department’s budget by 20%, and allocate it to customer support systems, our customer churn rate (attrition) will decrease by 5%.” Remember, your team also wants to address how the decreased marketing efforts impact the sales target for the next quarter. While customer attrition is critical and requires immediate resolution, decreased sales often affect your revenue and profits. Eventually, the reduced profits will upset some investors, affecting your stocks. This example is one of the several cases where companies require predictive analytics solutions for quick answers and data science consulting partners to gauge a business decision’s system-level (macro) impact. Conclusion The difference between predictive analytics solutions and data science models lies in their scale and complexity. Predictive analytics leverages basic statistics for trend discovery in structured datasets to plot the forecast curves. Data scientists require advanced statistical skillsets to extract valuable insights from the datasets, including semi-structured (infographics) and unstructured (video/audio) data. Moreover, you can only acquire realistic business insights with an accurately configured and trained machine learning model.

What Is a CDP (Customer Data Platform)? CDPs have risen up as one of the best solutions to tackle the challenge of data accessibility. Strictly speaking, CDPs collect and consolidate data from various sources and send that information to different target destinations (i.e., marketing tools and sales tools). The purpose of a CDP is to aggregate the information from various data sources and combine it together to create a single 360-degree view of the customer. In addition to this, they also provide an additional activation layer to enable marketing automation. This is because CDPs were created to analyze user behavior and personalize their experiences. Every company has data, so CDPs are useful for both B2C companies and B2B companies. Conventionally, it has been a huge challenge for marketers to gain access to data because all of the useful information is either stuck in disparate data sources and tools. CDPs solve this problem by supplying the marketing team with a relatively easy-to-use platform that requires little or no input from the data or engineering team. What Is Segment? Segment is one of the most popular CDPs. In fact, in 2020, Segment did over $144 million in revenue and was recently acquired by Twilio for $3.2 billion. At its core, Segment is a SaaS offering that helps businesses collect and leverage data from digital properties like websites, apps, SaaS tools, etc. Simply stated, Segment is an event-tracking platform that is aimed at app developers and SMB/mid-sized companies. Segment simplifies the data collection process and gives users the ability to spend more time leveraging their data to create personalized experiences and relevant content for customers. What Does Segment Do? Segment was originally created to solve the challenge of collecting and moving event data. In its simplest form, Segment helps fire user events captured in the product and sync that data to a variety of SaaS tools and data warehouses. It generates messages about what is happening in an app or website and then translates the information in those messages into a format that is understandable by other tools. Segment provides an API library that can run as code on a website, app, or server to generate messages based on specific triggers defined by the user. This code can be as simple as copying and pasting a snippet into the HTML of a website to track page views, or it can be embedded within an app to send messages when a user performs a specific action like opening or closing an app or abandoning a cart after a set amount of time. Once these messages have been generated, they can be sent directly to Segment servers to be translated or forwarded to specific destinations. Who Uses Segment? Segment has two core audiences, marketing teams and engineering teams. Segment appeals to marketers because it gives them an easy way to collect and merge different data sets together to create various customer profiles, enrich audiences, and activate campaigns across various tools. On the other hand, engineering teams are drawn to Segment because they don’t have to spend time writing their own event tracking library and writing integrations to all of their SaaS tools since all of this is supplied through Segment’s API library. This means engineers can focus their efforts on the high-priority tasks which have the most impact on a company’s bottom line. Best of all, marketers don’t have to go through the data or engineering team every time they want to ask a question or gain access to a specific data set because all of this is provided through Segment. What Are Segment Warehouses? Segment’s Warehouse feature gives users the ability to send information natively to various data warehouses (i.e. Snowflake, Amazon Redshift, Azure Synapse, Google BigQuery, etc.). This is super useful since the warehouse is typically the final resting point for data and acts as the analytics platform in most organizations. What Are Segment Personas? Segment Personas is a visual audience builder for marketers that gives businesses the ability to enrich customer profiles with new traits. Segment Personas take event data across multiple devices and channels and intelligently merge it together using identity resolution to create a single view of the customer. Segment defines an audience as either a list of users or accounts that match specific criteria. An example of this could be users who abandoned their shopping cart at X amount of time and purchased an item in the last seven days. These audiences are basically customized segments. Marketers can define these segments in a point-and-click UI without needing to know SQL. What Are Segment Functions? With Segment Functions, users can do basic transformations on events and send them to external tools and various APIs without having to set up or maintain any infrastructure. However, the transformation function is very limited and not nearly as strong as native languages like SQL or dedicated transformation tools like dbt. What Are the Alternatives to Segment? 1. mParticle mParticle is a Segment alternative. However, whereas Segment is tailored to SMB/mid-sized companies, mParticle is built for enterprise-sized companies. Instead of falling into the CDP category, mParticle brands itself as “Customer Data Infrastructure” or (CDI). At its core, CDI focuses on data integration, data governance, and audience management. Since mParticle is tailored toward enterprise companies, it places a higher focus on providing support. In fact, all mParticle customers are assigned a dedicated customer success rep on day one. Additionally, mParticle was actually one of the very first companies in the CDP space to offer professional services and release an audience-building product. To be specific, Segment’s Personas product was a direct result of mParticle’s Audiences product. Another core difference between the two lies in the fact that mParticle offers more robust capabilities around mobile event tracking (i.e. apps) and data integration. Since the Segment strictly tailors towards SMB, it really only focuses on web tracking. All in all, mParticle tends to offer more robust capabilities than Segment. Both solutions are tailored towards developers and have a substantial implementation/setup time before marketing teams can begin leveraging either tool to the fullest extent. 2. Tealium Tealium is a Segment alternative that places a higher emphasis on marketers instead of developers. Tealium is a CDP solution that focuses on enterprise-sized companies. Before becoming a fully-fledged CDP, Tealium fell into the category of “enterprise tag management” (i.e. a competitor to Google Tag Manager, which is a free service that gives users the ability to implement marketing tags or snippets of code for tracking purposes on their website). On the other hand, Tealium’s flagship product, “Tealium IQ”, offers more flexibility because it is not a native Google service like Google Tag Manager. This means it integrates with a variety of different platforms. Aside from offering the typical capabilities of a CDP, Tealium is HIPAA compliant and will sign BAAs or business associate agreements (a contract that outlines each party’s individual responsibilities for protected health information). The same cannot be said for either Segment or mParticle. Tealium’s main selling point is focused on privacy and security. This is why so many healthcare and financial services companies are drawn to it. This removes some of the 3rd party risks and other risk factors that can affect underlying business goals. 3. Lytics Lytics is an alternative to Segment that is very focused on empowering marketers rather than engineers and developers. Due to this reason, the implementation time for Lytics tends to be substantially longer than for players like Segment, mParticle, Tealium, etc. However, as an upside, Lytics has an extremely intuitive UI that is tailored toward marketers rather than developers, which makes it extremely streamlined and easy to use. Lytics has much more detailed and predictive machine learning capabilities compared to the other platforms. In fact, Lytics' Machine Learning API provides a framework to create custom ML models directly within the platform. These models are self-training and continuously update in real time. All audiences created in Lytics are also updated in real time with no user input. 4. RudderStack RudderStack is slightly different from the previous alternatives in that it is a fully open-source CDP platform tailored toward developers. RudderStack's core product functionality enables developers to deploy data pipelines and collect customer data from various apps, websites, and platforms to auto-track events. This information can then be activated in the data warehouse. Although RudderStack claims to be open-source, most of the features, like cloud connect, ETL/ELT, reverse ETL, etc., are locked behind the paid offering. RudderStack has a couple of upsides, though. Being an open-source platform, RudderStack is the only CDP that can run entirely on-premise. Additionally, Rudderstack does not own any of the data it hosts because everything is kept within an organization’s proprietary technology stack. In most cases, companies choose RudderStack when platforms like Segment get too expensive due to the MTU (monthly tracked user) pricing model in addition to the data ownership aspect. 5. SimonData SimonData is an email service platform combined with a CDP. It is very similar to solutions like Braze, Iterable, Salesforce Marketing Cloud, Marketo, etc. Most CDPs capture data from various sources to create audiences and then push that information back into operational platforms so that marketers can use it to launch campaigns. However, SimonData claims to do all of this in one. It connects natively to data warehouses, but it moves the data out of the warehouse, which can be very bad for compliance. SimonData also locks users into a simple user/event data model rather than supporting all types of data within the warehouse, like products, groups, flights, trips, purchases, etc. SimonData also creates another challenge in that it doesn’t support notifications efficiently. The needs of marketers are evolving extremely fast. This is one of the many reasons that companies are choosing to keep marketing platforms and data platforms separate and leverage dedicated solutions like Iterable or Braze-on top of a CDP. 6. ActionIQ ActionIQ focuses on helping companies achieve a full “digital transformation.” ActionIQ is very different from other CDPs because it leverages a database and adds a CDP as an additional layer on top. To be specific, ActionIQ helps companies assemble disparate data sources together into their own unique ActionIQ database and enables users to leverage this data through a conventional CDP. This solution tends to be very professional services-heavy, and getting data into the platform can be extremely challenging. It often takes up to a year to implement. Similarly, ActionIQ’s entire data model is focused solely on contacts and fields, so companies have little ability to leverage the data models that impact their business the most. It is really tailored towards businesses that have already made a significant investment into a specific technology stack and are simply looking for additional tools and data access. 7. Amperity Amperity’s core customers tend to be large retail or traditional brick-and-mortar businesses with extremely disparate data sources. Amperity is a CDP platform that is highly specialized in identity resolution. It has “state of the art” machine learning technologies, whereas most CDPs use simple “deterministic” identity resolution logic (e.g. static value equality on a graph, or if “email = email”, then this is the same user). Like most CDPs, Amperity does have some of the typical marketing activation capabilities that other CDPs offer. However, these are more limited. At its core, Amperity is extremely efficient at identifying and predicting customer behaviors which is a super useful trait for any company. What Are the Issues With Off-the-Shelf CDPs? All CDPs tend to have several similar problems. Firstly, CDPs are not a single source of truth. With the rise of cloud data warehouses (i.e. Snowflake, Redshift, BigQuery, Synapse, etc.), data warehouses now contain all customer data because companies are already using them for reporting and modeling. CDPs only have the data that is ingested into them. Secondly, CDPs create rifts in organizations because they were solely created for marketers. This discourages collaboration between marketing and data teams. Everyone within an organization needs to be working towards the same underlying goals, even if they are on different teams. Additionally, all CDPs are built on proprietary systems that don’t always pair well with other technologies. As an example, if a transformation capability doesn’t exist, users are stuck filing a support ticket. This actually happens quite frequently because conventional CDPs do not typically have the ability to do a ton of robust transformations on the data that is stored within them. Likewise, if an assortment of bad events is loaded into a CDP, users are limited to the features built-in to the CDP to clean that data set. Similarly, as a point-to-point tool that moves data back and forth between different systems, CDPs create silos because they cannot leverage any existing technologies or tools that may already exist in an organization. Additionally, since CDPs were created solely for marketers, the data models that they provide are not flexible. In fact, they often force organizations to “shoe-horn” their data into a strict model that makes no logical sense for the business. Lastly, CDPs store all of a company’s data which has privacy and security concerns. Each organization should own its own data so that it is not subject to the whims of a particular vendor. What Are the Issues With Segment? The segment does a decent job of moving data from point A to B. However, it has a couple of problems. Firstly, data that is pushed through Segment is never really transformed to create a proper 360 view of the customer (ex: combining billing and product data); it also cannot be combined with SQL. Segment claims to unify customers across all paths and channels to enable personalized campaigns, but these campaigns can only be so useful if all of the information that is being pushed to various marketing platforms is still in its raw state. Additionally, Segment’s data model is limited to two objects, users and accounts; and in most cases, a user can only belong to a single account. This is problematic because every business has a unique model. For example, a company like Spotify collects information on users and accounts, but it also tracks other concepts like artists and genres, which are typically treated as separate tables. However, the core problem with Segment is that it’s trying to take the place of a conventional iPaaS or ETL (extract, transform, load) tool and handle the entire end-to-end process of data integration. Additionally, with Twilio’s acquisition of Segment, it is safe to assume that there will be some bias in the tools that are recommended. After all, Twilio is focused entirely on contacting customers, and Segment is focused solely on managing the data about them. The segment does a good job of collecting and transferring event data. However, acquiring, ingesting, and transforming data from SaaS tools is another story. Using Segments is like renting a data pipeline, and most organizations want to control their technology stack from top to bottom. Proprietary data should be a competitive advantage and not a liability. Why Your Warehouse Should Be Your CDP The main difference between a CDP and a data warehouse lies in the fact that CDPs only store customer data, whereas data warehouses act as a repository for all the data across the entire organization - not just customer data. CDPs strictly focus on enriching data for marketing purposes, and data warehouses can run a variety of different workloads for analytics purposes. Most organizations have standardized the data warehouse as the single source of truth because the CDP only has a subset of data, whereas the warehouse has all of it. This is actually the number one reason why the data warehouse should be the CDP. Since all of the data is often already in the data warehouse, the logical choice is to simply just use it as a CDP. A modern data stack should consist of an end-to-end flow from data acquisition, collection, and transformation. In most cases, the easiest way to enable this goal is by leveraging tools that are purposely designed to handle a single task. Fivetran, Snowflake, and dbt are great examples of this. In fact, this is the core technology stack that every data-driven company is adopting. Fivetran handles the entire data integration aspect providing a simple SaaS solution that helps businesses quickly move data out of their SaaS tools and into their data warehouse. Snowflake provides an easy way for organizations to consolidate their data into one location for analytics purposes. Lastly, dbt provides a simple transformation tool that is SQL-based, enabling users to create data models that can be reused. These three solutions combined create an effective data management platform. However, there is a slight problem with this technology stack. Fivetran currently does not provide any way to collect and transfer event data (i.e. user actions). It also does not provide a way to move data out of the warehouse and back into the operational systems. This creates a problem because this is the main use case that Segment solves. How to Use Your Data Warehouse as Your CDP If Segment’s main advantage was solely the fact that it is able to collect event data and move that information to various SaaS platforms, this advantage is now gone, thanks to Reverse ETL tools and Behavioral Data Platforms. Solutions like these work unilaterally to create a more robust version of Segment by using the data warehouse as a CDP.

[Editor's note: Be sure to check out part 1, part 2 and part 3 first.] This is the fifth of a series of blogs introducing Apache HBase. In the fourth part, we saw the basics of using the Java API to interact with HBase to create tables, retrieve data by row key, and do table scans. This part will discuss how to design schemas in HBase. HBase has nothing similar to a rich query capability like SQL from relational databases. Instead, it forgoes this capability and others like relationships, joins, etc. to instead focus on providing scalability with good performance and fault-tolerance. So when working with HBase you need to design the row keys and table structure in terms of rows and column families to match the data access patterns of your application. This is completely opposite what you do with relational databases where you start out with a normalized database schema, separate tables, and then you use SQL to perform joins to combine data in the ways you need. With HBase you design your tables specific to how they will be accessed by applications, so you need to think much more up-front about how data is accessed. You are much closer to the bare metal with HBase than with relational databases which abstract implementation details and storage mechanisms. However, for applications needing to store massive amounts of data and have inherent scalability, performance characteristics and tolerance to server failures, the potential benefits can far outweigh the costs. In the last part on the Java API, I mentioned that when scanning data in HBase, the row key is critical since it is the primary means to restrict the rows scanned; there is nothing like a rich query like SQL as in relational databases. Typically you create a scan using start and stop row keys and optionally add filters to further restrict the rows and columns data returned. In order to have some flexibility when scanning, the row key should be designed to contain the information you need to find specific subsets of data. In the blog and people examples we've seen so far, the row keys were designed to allow scanning via the most common data access patterns. For the blogs, the row keys were simply the posting date. This would permit scans in ascending order of blog entries, which is probably not the most common way to view blogs; you'd rather see the most recent blogs first. So a better row key design would be to use a reverse order timestamp, which you can get using the formula (Long.MAX_VALUE - timestamp), so scans return the most recent blog posts first. This makes it easy to scan specific time ranges, for example to show all blogs in the past week or month, which is a typical way to navigate blog entries in web applications. For the people table examples, we used a composite row key composed of last name, first name, middle initial, and a (unique) person identifier to distinguish people with the exact same name, separated by dashes. For example, Brian M. Smith with identifier 12345 would have row key smith-brian-m-12345. Scans for the people table can then be composed using start and end rows designed to retrieve people with specific last names, last names starting with specific letter combinations, or people with the same last name and first name initial. For example, if you wanted to find people whose first name begins with B and last name is Smith you could use the start row key smith-b and stop row key smith-c (the start row key is inclusive while the stop row key is exclusive, so the stop key smith-c ensures all Smiths with first name starting with the letter "B" are included). You can see that HBase supports the notion of partial keys, meaning you do not need to know the exact key, to provide more flexibility creating appropriate scans. You can combine partial key scans with filters to retrieve only the specific data needed, thus optimizing data retrieval for the data access patterns specific to your application. So far the examples have involved only single tables containing one type of information and no related information. HBase does not have foreign key relationships like in relational databases, but because it supports rows having up to millions of columns, one way to design tables in HBase is to encapsulate related information in the same row - a "wide" table design. It is called a "wide" design since you are storing all information related to a row together in as many columns as there are data items. In our blog example, you might want to store comments for each blog. The "wide" way to design this would be to include a column family named comments and then add columns to the comment family where the qualifiers are the comment timestamp; the comment columns would look like comments:20130704142510 and comments:20130707163045. Even better, when HBase retrieves columns it returns them in sorted order, just like row keys. So in order to display a blog entry and its comments, you can retrieve all the data from one row by asking for the content, info, and comments column families. You could also add a filter to retrieve only a specific number of comments, adding pagination to them. The people table column families could also be redesigned to store contact information such as separate addresses, phone numbers, and email addresses in column families allowing all of a person's information to be stored in one row. This kind of design can work well if the number of columns is relatively modest, as blog comments and a person's contact information would be. If instead you are modeling something like an email inbox, financial transactions, or massive amounts of automatically collected sensor data, you might choose instead to spread a user's emails, transactions, or sensor readings across multiple rows (a "tall" design) and design the row keys to allow efficient scanning and pagination. For an inbox the row key might look like <user_id>-<reversed_email_timestamp> which would permit easily scanning and paginating a user's inbox, while for financial transactions the row key might be <user_id>-<reversed_transaction_timestamp>. This kind of design can be called "tall" since you are spreading information about the same thing (e.g. readings from the same sensor, transactions in an account) across multiple rows, and is something to consider if there will be an ever-expanding amount of information, as would be the case in a scenario involving data collection from a huge network of sensors. Designing row keys and table structures in HBase is a key part of working with HBase, and will continue to be given the fundamental architecture of HBase. There are other things you can do to add alternative schemes for data access within HBase. For example, you could implement full-text searching via Apache Lucene either within rows or external to HBase (search Google for HBASE-3529). You can also create (and maintain) secondary indexes to permit alternate row key schemes for tables; for example in our people table the composite row key consists of the name and a unique identifier. But if we desire to access people by their birth date, telephone area code, email address, or any other number of ways, we could add secondary indexes to enable that form of interaction. Note, however, that adding secondary indexes is not something to be taken lightly; every time you write to the "main" table (e.g. people) you will need to also update all the secondary indexes! (Yes, this is something that relational databases do very well, but remember that HBase is designed to accomodate a lot more data than traditional RDBMSs were.) Conclusion to Part 5 In this part of the series, we got an introduction to schema design in HBase (without relations or SQL). Even though HBase is missing some of the features found in traditional RDBMS systems such as foreign keys and referential integrity, multi-row transactions, multiple indexes, and son on, many applications that need inherent HBase benefits like scaling can benefit from using HBase. As with anything complex, there are tradeoffs to be made. In the case of HBase, you are giving up some richness in schema design and query flexibility, but you gain the ability to scale to massive amounts of data by (more or less) simply adding additional servers to your cluster. In the next and last part of this series, we'll wrap up and mention a few (of the many) things we didn't cover in these introductory blogs. References HBase web site, http://hbase.apache.org/ HBase wiki, http://wiki.apache.org/hadoop/Hbase HBase Reference Guide http://hbase.apache.org/book/book.html HBase: The Definitive Guide, http://bit.ly/hbase-definitive-guide Google Bigtable Paper, http://labs.google.com/papers/bigtable.html Hadoop web site, http://hadoop.apache.org/ Hadoop: The Definitive Guide, http://bit.ly/hadoop-definitive-guide Fallacies of Distributed Computing, http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing HBase lightning talk slides, http://www.slideshare.net/scottleber/hbase-lightningtalk Sample code, https://github.com/sleberknight/basic-hbase-examples

The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems. Storing data at rest for reporting and analytics requires different capabilities and SLAs than continuously processing data in motion for real-time workloads. Many open-source frameworks, commercial products, and SaaS cloud services exist. Unfortunately, the underlying technologies are often misunderstood and overused for monolithic and inflexible architectures, and vendors pitch for the wrong use cases. Let’s explore this dilemma in a blog series. Learn how to build a modern data stack with cloud-native technologies. Blog Series: Data Warehouse vs. Data Lake vs. Data Streaming This blog series explores concepts, features, and trade-offs of a modern data stack using a data warehouse, data lake, and data streaming together: Data Warehouse vs. Data Lake vs. Data Streaming - Friends, Enemies, Frenemies? THIS POST: Data Streaming for Data Ingestion into the Data Warehouse and Data Lake Data Warehouse Modernization: From Legacy On-Premise to Cloud-Native Infrastructure Case Studies: Cloud-native Data Streaming for Data Warehouse Modernization Lessons Learned from Building a Cloud-Native Data Warehouse Stay tuned for a dedicated blog post for each topic as part of this blog series. I will link the blogs here as soon as they are available (in the next few weeks). Reliable and Scalable Data Ingestion With Data Streaming Reliable and scalable data ingestion is crucial for any analytics platform, whether you build a data warehouse, data lake, or lakehouse. Almost every major player in the analytics space re-engineered the existing platform to enable data ingestion in (near) real-time instead of just batch integration. Data Ingestion = Data Integration Just getting data from A to B is usually not enough. Data integration with Extract, Transform, Load (ETL) workflows connects to various source systems and processes incoming events before ingesting them into one or more data sinks. However, most people think about a message queue when discussing data ingestion. Instead, a good data ingestion technology provides additional capabilities: Connectivity: Connectors enable a quick but reliable integration with any data source and data sink. Don't build your own connector if a connector is available! Data processing: Integration logic filters, enriches or aggregates data from one or more data sources. Data sharing: Replicate data across regions, data centers, and multi-cloud. Cost-efficient storage: Truly decouple data sources from various downstream consumers (analytics platforms, business applications, SaaS, etc.) Kafka is more than just data ingestion or a message queue. Kafka is a cloud-native middleware: One common option is to run ETL workloads in the data warehouse or data lake. A message queue is sufficient for data ingestion. Even in that case, most projects use the data streaming platform Apache Kafka as the data ingestion layer, even though many great Kafka features are disregarded this way. Instead, running ETL workloads in the data ingestion pipeline has several advantages: True decoupling between the processing and storage layer enables a scalable and cost-efficient infrastructure. ETL workloads are processed in real-time, no matter what pace each downstream application provides. Each consumer (data warehouse, data lake, NoSQL database, message queue) has the freedom of choice for the technology/API/programming language/communication paradigm to decide how and when to read, process, and store data. Downstream consumers choose between raw data and curated relevant data sets. Data Integration at Rest or in Motion? ETL is nothing new. Tools from Informatica, TIBCO, and other vendors provide great visual coding for building data pipelines. The evolution in the cloud brought us cloud-native alternatives like SnapLogic or Boomi. API Management tools like MuleSoft are another common option for building an integration layer based on APIs and point-to-point integration. The enormous benefit of these tools is a much better time-to-market for building and maintaining pipelines. A few drawbacks exist, though: Separate infrastructure to operate and pay for. Often limited scalability and/or latency using a dedicated middleware compared to a native stream ingestion layer (like Kafka or Kinesis). Data stored at rest in separate storage systems. Tight coupling with web services and point-to-point connections. Integration logic lies in the middleware platform, expertise, and proprietary tooling without freedom of choice for implementing data integration. The great news is that you have the freedom of choice. If the heart of the infrastructure is real-time and scalable, then you can add not just scalable real-time applications but any batch or API-based middleware: Point-to-point integration via API Management is complementary to data streaming, not competitive. Data Mesh for Multi-Cloud Data Ingestion Data Mesh is the latest buzzword in the software industry. It combines domain-driven design, data marts, microservices, data streaming, and other concepts. Like Lakehouse, Data Mesh is a logical concept, NOT physical infrastructure built with a single technology! I explored what role data streaming with Apache Kafka plays in a data mesh architecture. In summary, the characteristics of Kafka, such as true decoupling, backpressure handling, data processing, and connectivity to real-time and non-real-time systems, enable the building of a distributed global data architecture. Here is an example of a data mesh built with Confluent Cloud and Databricks across cloud providers and regions: Data streaming as the data ingestion layer plays many roles in a multi-cloud and/or multi-region environment: Data replication between clouds and/or regions. Data integration with various source applications (either directly or via 3rd party ETL/iPaaS middleware). Preprocessing in the local region to reduce data transfer costs and improve latency between regions. Data ingestion into one or more data warehouses and/or data lakes. Backpressure handling for slow consumers or between regions in case of disconnected internet. The power of data streaming becomes more apparent in this example: Data ingestion or a message queue alone is NOT good enough for most "data ingestion projects"! Apache Kafka — The de Facto Standard for Data Ingestion Into the Lakehouse Apache Kafka is the de facto standard for data streaming, and a critical use case is data ingestion. Kafka Connect enables reliable integration in real-time at any scale. It automatically handles failure, network issues, downtime, and other operations issues. Search for your favorite analytics platform and check the availability of a Kafka Connect connector. The chances are high that you will find one. Critical advantages of Apache Kafka compared to other data ingestion engines like AWS Kinesis include: True decoupling: Data ingestion into a data warehouse is usually only one of the data sinks. Most enterprises ingest data into various systems and build new real-time applications with the same Kafka infrastructure and APIs. Open API across multi-cloud and hybrid environments: Kafka can be deployed everywhere, including any cloud provider, data center, or edge infrastructure. Cost-efficiency: The more you scale, the more cost-efficient Kafka workloads become compared to Kinesis and similar tools. Long-term storage in the streaming platform: Replayability of historical data and backpressure handling for slow consumers are built into Kafka (and cost-efficient if you leverage Tiered Storage). Pre-processing and ETL in a single infrastructure: Instead of ingesting vast volumes of raw data into various systems at rest, the incoming data can be processed in motion with tools like Kafka Streams or ksqlDB once. Each consumer chooses the curated or raw data it needs for further analytics - in real-time, near real-time batch, or request-response. Kafka as Data Ingestion and ETL Middleware With Kafka Connect and KsqlDB Kafka provides two capabilities that many people often underestimate in the beginning (because they think about Kafka just as a message queue): Data integration: Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other data systems. Kafka Connect makes it simple to quickly define connectors that move large data sets into and out of Kafka. Stream Processing: Kafka Streams is a client library for building stateless or stateful applications and microservices, where the input and output data are stored in Kafka clusters. ksqlDB is built on top of Kafka Streams to build data streaming applications leveraging your familiarity with relational databases and SQL. The key difference from other ETL tools is that Kafka eats its own dog food: Please note that Kafka Connect is more than a set of connectors. The underlying framework provides many additional middleware features. Look at single message transforms (SMT), dead letter queue, schema validation, and other capabilities of Kafka Connect. Kafka as Cloud-Native Middleware; NOT iPaaS Data streaming with Kafka often replace ETL tools and ESBs. Here are the key reasons: Some consider data streaming an integration platform as a service (iPaaS). I can't entirely agree with that. The arguments are very similar to saying Kafka is an ETL tool. These very different technologies shine with other characteristics (and trade-offs). Reference Architectures for Data Ingestion With Apache Kafka Let's explore two example architectures for data ingestion with Apache Kafka and Kafka Connect: Elasticsearch Data Streams and Databricks Delta Lake. Both products serve very different use cases and require a different data ingestion strategy. The developer can use the same Kafka Connect APIs for different connectors. Under the hood, the implementation looks very different to serve different needs and SLAs. Both use the dedicated Kafka Connect connector on the high-level architecture diagram. Under the hood, ingestion speed, indexing strategy, delivery guarantees, and many other factors must be configured depending on the project requirements and data sink capabilities. Data Ingestion Example: Real-time Indexing With Kafka and Elasticsearch Data Streams One of my favorite examples is Elasticsearch. Naturally, the search engine built indices in batch mode. Hence, the first Kafka Connect connector ingested events in batch mode. However, Elastic created a new real-time indexing strategy called Elasticsearch Data Streams to offer its end users faster query and response times. An Elastic Data Stream lets you store append-only time series data across multiple indices while giving you a single named resource for requests. Data streams are well-suited for logs, events, metrics, and other continuously generated data. You can submit indexing and search requests directly to a data stream. The stream automatically routes the request to backing indices that store the stream’s data. A Kafka data stream is the ideal data source for Elastic Data Streams. Data Ingestion Example: Confluent and Databricks Confluent, in conjunction with Databricks, is another excellent example. Many people struggle to explain both vendors' differences and unique selling points. Why? Because the marketing looks very similar for both: Process big data, store it forever, analyze it in real-time, deploy across multi-cloud and multi-region, and so on. The reality is that Confluent and Databricks overlap only ~10%. Both complement each other very well. Confluent's focus is to process data in motion. Databricks' primary business is storing data at rest for analytics and reporting. You can store long-term data in Kafka (especially with Tiered Storage). Yes, you can process data with Databricks in (near) real-time with Spark Streaming. That's fine for some use cases, but in most scenarios, you (should) choose and combine the best technology for a problem. Here is a reference architecture for data streaming and analytics with Confluent and Databricks: Confluent and Databricks are a perfect combination of a modern machine learning architecture: Data ingestion with data streaming. Model training in the data lake. (streaming or batch) ETL where it fits the use case best. Model deployment in the data lake close to the data science environment or a data streaming app for mission-critical SLAs and low latency. Monitoring the end-to-end pipeline (ETL, model scoring, etc.) in real-time with data streaming. I hope it is clear that data streaming vendors like Confluent have very different capabilities, strengths, and weaknesses than data warehouse vendors like Databricks or Snowflake. How to Build a Cloud-Native Lakehouse With Kafka and Spark? Databricks coined Lakehouse to talk about real-time data in motion and batch workloads at rest in a single platform. From my point of view, the Lakehouse is a great logical view. But there is no silver bullet! The technical implementation of a Lakehouse requires different tools in a modern data stack. In June 2022, I presented my detailed perspective on this discussion at Databricks’ Data + AI Summit in San Francisco. Check out my slide deck "Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture". Data Streaming Is Much More Than Data Ingestion Into a Lakehouse Data streaming technologies like Apache Kafka are perfect for data ingestion into one or more data warehouses and/or data lakes. BUT data streaming is much more: Integration with various data sources, data processing, replication across regions or clouds, and finally, data ingestion into the data sinks. Examples with vendors like Confluent, Databricks, and Elasticsearch showed how data streaming helps solve many data integration challenges with a single technology. Nevertheless, there is no silver bullet. A modern lakehouse leverages a best of breed technologies. No single technology can be the best to handle all kinds of data sets and communication paradigms (like real-time, batch, and request-response). For more details, browse other posts of this blog series: Data Warehouse vs. Data Lake vs. Data Streaming - Friends, Enemies, Frenemies? THIS POST: Data Streaming for Data Ingestion into the Data Warehouse and Data Lake Data Warehouse Modernization: From Legacy On-Premise to Cloud-Native Infrastructure Case Studies: Cloud-native Data Streaming for Data Warehouse Modernization Lessons Learned from Building a Cloud-Native Data Warehouse How do you combine data warehouse and data streaming today? Is Kafka just your ingestion layer into the data lake? Do you already leverage data streaming for additional real-time use cases? Or is Kafka already the strategic component in the enterprise architecture for decoupled microservices and a data mesh? Stay Tune.

Among the open-source projects my college buddies (and my future co-founders of memphis.dev) and I built, you can find “Makhela,” a Hebrew word for choir. For the sake of simplicity, we will call it "Choir" in this article. “Choir” was an open-source OSINT (Open-Source Intelligent) project focused on gathering context-based connections between social profiles using AI models like LDA and topic modeling, written in Python to explain what the world discusses over a specific domain and by high-ranking influencers in that domain and to focus on what's going on at the margins. For proof-of-concept or MVP, we used a single data source, fairly easy to integrate: Twitter. The graph below shows the “brain” behind “Choir.” The brain autonomously grows and analyzes new vertexes and edges based on incremental changes in the corpus and fresh ingested data. Each vertex symbolizes a profile, a persona, and each edge emphasizes (a) who connects to who. (b) Similar color = Similar topic. Purple = Topic 1Blue = Topic 2Yellow = Marginal topic After a reasonable amount of research, dev time, and a lot of troubleshooting and debugging, things started to look good. Among the issues we needed to solve were the following: Understand the connection between profiles. Build a ranking algorithm for adding more influencers. Transform the schema of incoming data to a shape the analysis side knows how to handle. Near real-time is crucial: Enrich each tweet with external data. Adaptivity to "Twitter" rate limit. Each upstream or schema change crashed the analysis functions. Sync between collection and analysis, which were two different components. Infrastructure Scale As with any startup or early-stage project, we built “Choir” as MVP working solely with “Twitter,” and it looked like this: The “Collector” is a monolith, Python-written application that basically collects and refines the data for analysis and visualization in batches and in a static timing every couple of hours. Problems started to arise as the collected data, and its complexity grew. The amount of processing cycle each batch processing took to be analyzed turned into hours for no good reason in terms of the capacity of the collected data (hundreds of megabytes at most!)—more on the rest of the challenges in the next sections. Fast forward a few months later, users started to use “Choir!” Not just using but actually engaging, paying, and raising feature requests. Any creator’s dream! But then it hit us. (a) "Twitter" is not the center of the universe, and we need to expand “Choir” to more sources. (b) Any minor change in the code breaks the entire pipeline. (c) Monolith is a death sentence to a data-driven app, performance-wise. As with every eager-to-run project that starts to get good traction, fueling that growth and user base is your number 1, 2, and 3 priority, and the last thing you want to do at this point is to go back and rebuild your framework. You want to continue the momentum. With that spirit in our minds, we said, "Let's add more data sources and refactor in the future." Big mistake. Challenges in Scaling a Data-driven Application Each new data source requires a different schema transformation. Each schema change causes a chain of reactions downstream to the rest of the stages in the pipeline. Incremental/climbing collection: While you can wait for an entire batch collection to finalize and then save it to the DB, applications often get crashed. Imagine you're doing a very slow collection, and in the very last record, the collection process gets crashed. In a monolith architecture, it's hard to scale out the specific functions that require more power. Analysis functions often require modifications, upgrades, and algorithms to get better results, which are made by using or requiring different keys from the collectors. While there is no quick fix, what we can do is build a framework to support such requirements. Solutions Option 1 Duplicate the entire existing process to another source, for example, “Facebook.” Besides duplicating the collector, we needed to: Maintain two different schemas (Nightmare). Entirely different analysis functions. The connections between profiles on "Facebook" and "Twitter" are different and require different objective relationships. The analyzer should be able to analyze the data in a joined manner, not individually; therefore, any minor change in source X directly affects the analyzer and often crashes it down. Double maintenance And the list goes on… Long story short, it can't scale. Option 2 Here it comes. Use a message broker! I want to draw a baseline. A message broker is not the solution but a supporting framework or a tool to enable branched, growing data-driven architectures. What Is a Message Broker? According to Wikipedia, “A message broker is an architectural pattern for message validation, transformation, and routing. It mediates communication among applications, minimizing the mutual awareness that applications should have of each other in order to be able to exchange messages, effectively implementing decoupling.” Let’s translate it to something we can grasp better. A message broker is a temporary data store. Why temporary? Because each piece of data within it will be removed after a certain time, defined by the user. The pieces of data within the message broker are called “messages.” Each message usually weighs a few bytes to a few megabytes. Around the message broker, we can find producers and consumers. Producer = The “thing” that pushes the messages into the message broker.Consumer = The “thing” that consumes the messages from the message broker. “Thing” means system/service/application/IoT/some objective that connects with the message broker and exchanges data. Small note: the same service/system/app can act as a producer and consumer at the same time. Messaging queues derive from the same family, but there is a crucial difference between a broker and a queue. MQ uses the term publish and subscribe. The MQ itself pushes the data to the consumers and not the other way (the consumer pulls data from the broker). Ordering is promised. Messages will be pushed in the order they receive. Some systems require it. The ratio between a publisher (producers) and subscribers is 1:1. Having said it, modern versions can achieve it through some features like exchange and more. Some well-known message brokers/queues include Apache Kafka, RabbitMQ, Apache Pulsar, and our own. Still with me? Awesome! Let’s understand how using a message broker helped “Choir” to scale. Instead of doing things like this: By decoupling the app into smaller microservices and orchestrating the flow using a message broker, it turned into this: Starting from the top-left corner, each piece of data (tweet/post) inserted into the system automatically triggers the entire process and flows between the different stages. Collection: The three collectors search for each new profile added to the community in parallel. If any additional data source/social network is needed, it's been developed on the side — and once ready, start listening for incoming events. The collection allows an infinite scale of sources, the ability to work on a specific source without disrupting the others, micro-scaling for better performance of each source individually, and more. Transformation: Once the collection is complete, results will be pushed to the next stage, “schema transformation,” where the schema transformation service will transform the events’ schemas into a shape the analysis function can interpret. It enables a “single source of truth” regarding schema management, so in case of upstream change, all is needed to reach out to this service and debug the issue. In a more robust design, it can also integrate with an external schema registry to make maintenance even more effortless. Analysis: Each piece of event is sent to the analysis function transformed, and in shape, the analysis function can interpret. In “Choir,” we used different AI models. Scaling it was impossible, so moving to analysis per event definitely helped. Save: Creates an abstraction between “Choir” and the type of database + ability to batch several insertions to a single batch instead of request per event. The main reason behind my writing is to emphasize the importance of implementing a message broker pattern and technology as early as possible to avoid painful refactoring in the future. Yes, your roadmap and added features are important. Yes, it will take a learning curve. Yes, it might look like an overkill solution for your stage — but when it comes to a data-driven use case, the need for scale will reveal itself quickly in performance, agility, feature additions, modifications, and more. Bad design decisions or a lack of proper framework will burn out your resources. It is better to build agile foundations, not necessarily enterprise-grade, before reaching the phase where you are overwhelmed by users and feature requests. To conclude, the entry barrier for a message broker is definitely worth your time.

A semantic layer is something we use every day. We build dashboards with yearly and monthly aggregations. We design dimensions for drilling down reports by region, product, or whatever metrics we are interested in. What has changed is that we no longer use a singular business intelligence tool; different teams use different visualizations (BI, notebooks, and embedded analytics). Instead of re-creating siloed metrics in each app, we want to define them once, open in a version-controlled way and sync them into each visualization tool. That’s what the semantic layer does, primarily defined as YAML. Additionally, the semantic layer adds powerful layers such as APIs, caching, access control, data modeling, and metrics layer. ℹ️ The metrics layer is one component of a semantic layer. A limited metrics layer is usually built into a BI tool, translating its metrics to only that BI tool. The metrics and semantic layers we’ll talk about in this article support multiple BI tools, notebooks, and data apps. ℹ️ The semantic layer, metrics layer, headless BI, and sometimes even metrics store are similar. Although the metrics layer is a part of the semantic layer, I will use them as synonyms and refer to all of them as a semantic layer, as the differences are minor and will lead to better understanding. If you like to dig more into the details, check out the glossary definitions of a Semantic Layer, Metrics Layer, and Headless BI. What Is a Semantic Layer? A semantic layer is a translation layer that sits between your data and your business users. The semantic layer converts complex data into understandable business concepts. For example, your database may store millions of sales receipts that contain information such as sale amount, sale location, time of sale, etc. However, your business users may not have the technical capabilities to convert such raw data into actionable insights such as revenue per store, which brand is most popular, etc. By translating business terms into a format that is understood by the underlying database, the Semantic Layer allows business users to access data using terms that they are familiar with. You can think of a semantic layer as a translation layer between any data presentation layer (BI, notebooks, data apps) and the data sources. A translation layer includes many features, such as integrating data sources, modeling the metrics, and integrating with the data consumers by translating metrics into SQL, REST, or GraphQL. Because everyone has different definitions of “active” users or “paying” customers, the semantic layer allows you to define these discrepancies once company-wide. Describing each of them earlier in the semantic layer, instead of three different versions in each BI tool, avoids having contrasting numbers in various tools. And what if the metric changes to a new definition? With a semantic layer, you change only once. This powerful feature empowers domain experts and data practitioners to get a common understanding of business metrics. The 101 on Metrics, Measure, and Dimension Metric, KPI, and (calculated) measure are terms that serve as the building blocks for how business performance is both measured and defined, as knowledge of how to define an organization's KPIs. It is fundamental to have a common understanding of them. Metrics usually surface as business reports and dashboards with direct access to the entire organization.For example, think of operational metrics that represent your company's performance and service level or financial metrics that describe its financial health. Today these metrics are primarily defined in a lengthy SQL statement inside the BI tools.Calculated measures are part of metrics and apply to specific dimensions traditionally mapped inside a Bus Matrix. Dimensions are the categorical buckets that can be used to segment, filter, group, slice, and dice—such as sales amount, region, city, product, color, and distribution channel. Dimensions (and facts) are also known from the concept of Dimensional Modeling. Let’s see an example of how we define metrics such as “amount of users” and “paying users” and related dimensions city and company (more on Cube Data Schema). At its core, the semantic layer takes a metric request as input: JSON cube(`Users`, { sql: `SELECT * FROM users`, measures: { count: { sql: `id`, type: `count`, }, payingCount: { sql: `id`, type: `count`, filters: [{ sql: `${CUBE}.paying = 'true'` }], }, }, dimensions: { city: { sql: `city`, type: `string`, }, companyName: { sql: `company_name`, type: `string`, }, }, }); and returns a translated SQL query of its destination interpreter (mostly SQL, REST, GraphQL) and extracts those metrics as an output: SQL SELECT count( CASE WHEN (users.paying = 'true') THEN users.id END ) "users.paying_count" FROM users In new semantic layers, key business metrics are defined non-repetitively by the business and data teams as code (YAML). The goal is to abstract metrics so business people can discuss with data engineers and transfer ownership closer to the domain experts. The metrics are higher-level abstractions that are stored in a central repository in a declarative manner. The most famous existing semantic layer is LookML from Looker. The difference is that we separate that part of all BI tools and centralize them. Doing so allows us to define them once in a structured and source-controlled way instead of inside a proprietary tool. Understanding Transformation Layer vs Semantic Layer The transformation logic (e.g., dbt) and logic hosted in metrics are different. A semantic layer transforms/joins data at query time, whereas the transformation layer does during the transform (T) step of ETL/ELT, meaning it’s pre-calculated. In the transformation layer, you must balance low and high granularity. What level do you aggregate and store (e.g., rollups hourly data to daily to save storage), or what valuable dimensions to add. With each dimension and its column added, rows will explode exponentially, and we can’t persist each of these representations to the filesystem. A semantic layer is much more flexible and makes the most sense on top of transformed data in a data warehouse. Avoid extensive reshuffles or reprocesses of large amounts of data. Think of OLAP cubes where you can dice-and-slice ad-hoc on significant amounts of data without storing them ahead of time. Suppose we would push all metric logic into the transformation layer, with dbt, for example. In that case, we’d lose this agility and store multiple versions of each table with different granularities. Reprocessing when quickly changing a metric would result in long ETL jobs not finishing overnight. Deep Dive Into a Semantic Layers Architecture As you can see in the illustration below, the semantic layer sits on top of your data storage, mostly after your transformations are done. In smaller data use cases, you could use it on top of your data storage without transforming and do everything logically in the semantic layer. It's also where data engineers and business/analytics people need to find a standard metric definition before moving metrics downstream to the presentation layer. Semantic Layer integrated with today's Data Architecture An essential question is where the queries are executed. A semantic layer does not include a compute per se – you can substitute that with Kubernetes, other compute engines, or respective cloud offerings of semantic layers. Still, your semantic layers use computing where data lives – push-down. Push-downs are more difficult if you join multiple sources, but it's not a problem if you only pull from your warehouse. Recapping: Managing today's data ecosystem – and gaining valuable business insights – is a challenge for all companies, and that’s where the semantic layer helps: It builds bridges between business and (data) engineers. A time-saving solution for accessing a specific data set on top of your data stack. It centralizes business logic definitions that everyone agrees with, having the DRY principle applied across BI tools and the data stack. Key metrics can be easily tracked. The semantic layer improves data warehouse performance by taking on the processing burden to meet the analyst's needs. Moreover, it significantly reduces analytics processing costs. It offers an easy entry, as it can be the start of a future enterprise data warehouse project. What a Semantic Layer Is Not Before we cover more on the history and trends of a semantic layer, let’s quickly cover what it is not. It is not a replacement for the transformation layer or BI tools not an all-in-one solution that replaces everything not an OLAP cube The History of a Semantic Layer Designing a Semantic Layer in SAP BusinessObject Universe Designer | YouTube The semantic layer is not something entirely new. The concept existed since 1991 when SAP Business Objects (BO) patented it. As an early Business Intelligence engineer, I used lots of BI tools. I was always impressed by the BO universe and how you could abstract complex queries and models away from the business users, and design it to their understanding of the world. The basic or key component of business intelligence was always a semantic layer. It allowed an abstraction or translation from technical terms to business terms, such as facts and dimensions, and mapped it to more understandable models for the business. But it was always part of the BI Tools or the query or reporting layer. Later, when we had OLAP Cubes, such as the famous SSAS one, they also had their own semantic layer called data source view or cube structure, where you defined relations, measures, dimensions, and calculated measures. Also, remember the famous Bus Matrix, where you define measures to their dimensions. SSAS Dimensions and Cube Basics | Exsilio Blog The SAP BO Universe was a semantic layer that closed the gap between technology and business with an intermediate layer. You could define logical objects on top of physical ones, basically a CTEs at query time. Similar, but not a semantic layer, Master Data Management (MDM) came up later and tried to ensure master data (e.g., customer, product, address) is consistent, accurate, and maintained once with semantic consistency. Giving accountability with stewardship to the business users. MDM’s most popular tool I used was SQL Master Data Services (MDS) by Microsoft, which was announced in 2008 and updated last in 2015. Around the same time (July 2008), as MDS was announced, another tool called Jinja Template was released for the first time. It is not a semantic layer, but it helps define complex business logic within Python or SQL code. The Jinja template didn’t get much popularity inside the data world until dbt was announced and used heavily to mitigate the downside of SQL, creating complex models. dbt was originally started at RJMetrics in 2016 but took off later in 2018 when Fishtown Analytics (now called dbt Labs) more publicly showed it, and the market was more ready for it. People built huge semantic business layers on top of dbt and Jinja templates but lacked many features, such as ad-hoc definition and a declarative approach–dbt is imperative. You need to run a dbt run to get anything. Besides other BI tools having similar modeling language to define metrics, Looker, with its LookML language, popularized the modern semantic layer methodology. It wasn’t called such, but people saw its benefit and used Looker heavily. Later, when data grew exponentially, and companies didn’t use a single BI tool any longer, the need for transferring these metrics and calculated business logic to other tools such as Notebooks and excel sheets grew. In 2021, the “new” modern semantic layer rose in popularity with tools MetriQL, Minerva by Airbnb, MetricFlow, Supergrain, or Cube.js. The Evolution of the Semantic Layer1991: SAP BusinessObjects Universe and BI semantic layer 2008: Master Data Management (MDM) (with MDS from Microsoft in 2008) 2013: Kimball discussed the concept of a semantic layer in #158 Making Sense of the Semantic Layer 2016: Maturing BI tools with an integrated semantic layer such as Tableau, TARGIT, PowerBI, Apache Superset, etc. have their own metrics layer definition 2018: Jinja templates and dbt eroding the transformation layer into a semantic layer 2019: Looker and LookML popularized as the first real semantic layer 2022: Modern Semantic Layer, Metric Layer or Headless BI tools such as MetriQL, MetricFlow, Minerva, dbt arose with the explosion of data tools (BI tools, notebooks, spreadsheets, machine learning models, data apps, reverse ETL, …) The Rise of the Semantic Layer Let’s talk about why the data world is picking up the semantic layer again and where it fits into the data and AI landscape and the modern data stack. A Google Trends Search over the last 10+ years Several high-level reasons explain the rise of the semantic layer: The need for abstractions for complex business constructs and creating bridges between business people and engineers at a centralized place with code. Reduction of metric duplications eases the understanding and agreement on metrics. The demand for data lineage also contributes to the metrics and semantic layer. It’s the first place where data consumers want to know where the data comes from. The semantic layer is a core component between existing tools and rising concepts such as OLAP cubes, data virtualizations, data catalogs or data mesh, and data contracts. Diving into more details of each bullet point above. With the fast-rising complexity of data and its ecosystem, you also need better abstractions! They are lowering the bar for people to enter the field by easing up the process and complex data world. We’ve seen this with React for websites, Terraform for Kubernetes, and Dagster for orchestration. But yet, we miss one for metrics. That’s where the interest in the semantic layer comes from. With it, we abstract complex business requirements into smaller and easier-to-handle blocks. With open-source on the rise, it was natural that the next semantic layer would be open-source. Unlike Business Objects in the old days, see Transform that started closed-source but ended up open-sourcing part of it with MetricFlow. With metrics and the semantic layer rising as a core component, we want to have this layer open-sourced, avoiding the old trap of siloed data. With abstractions and open-source, we also ride the wave of the declarative approach. What does that mean? It means predefined data types and syntax for data modeling, such as how to create a distinct count measure or define a dimension. It also means that you can describe your metrics in a declarative way, e.g., YAML, without running anything ahead of time. The framework is more supportive with pre-defined abstractions, e.g., the most complex profit calculation is just one variable you can reference. By abstracting and making it declarative, it will be less error-prone, with less complexity and code duplication. If we look at the modern data stack and how the data flows, we see many ways to improve or intercept metrics along the way: Managed ETL and ELT that integrates data from various data sources Data storage such as data warehouse or data lake Data transformation layer that processes ingested data using SQL, Python, or YAML Different BI dashboards, notebooks, machine learning models, or data apps for visualization Data governance and data quality processes There is a lot of business logic involved in each step. For example, you clean up NULLs or fix improper formatting of dates in step one. You might add serverless functions on top of data warehouses or lakes to transform data but certainly, add a lot of data modeling and transformation in your transformation layer. And even more metrics and complex business calculations in the relevant BI tools. Plus, some cleaning up is happening in the data quality and governance. This example illustrates that it’s hard to sync these different logics and even different languages into one solution. That’s what the semantic layer try to solve holistically, making it so powerful. There are also big problems with such an approach, which we will explore in a later chapter. But another benefit you get from it is data lineage. As you have a semantic layer that explains many of your business logic declarative ways, you can also express where the source is and how it’s been transformed in your data. The more you have end-to-end, you can also do column-level lineage. This lineage is a huge step toward a more self-serve approach we all try to create. It’s also a way to move closer to the source data and the transformation layer. ℹ️ What others are sayingMaxime Beauchemin sees similar trends in his article: "It is clear that there’s huge value in these semantics and that the cost of maintaining this layer is substantial. Generally, the trend is for these semantics to move closer or into the transform layer, be progressively adaptable, be managed in source control as opposed to GUIs, and the need for it to become less siloed/duplicated.“ Trends: What Tools Are Out There for Building a Semantic Layer? As we’ve now learned a lot about the semantic layer, let's see actual tools, focusing on open-source. The most commonly named tools are Cube.js (recently renamed Cube), MetricFlow, MetriQL, dbt Metrics, or Malloy. Where MetriQL was the first open-source but Transform.co working on a closed-source version that eventually got open-sourced into MetricFlow. Also, dbt announced their metrics system back at Coalesce the Metric System, and turning it now into dbt Semantic Layer (more to come in October at dbt Coalesce Conference). According to GitHub stars, Cube is the fastest growing tool in this area, and already has many integrations including data sources and data visualizations – for example, it integrates with dbt Metrics. MetricFlow and Cube’s semantic layer overview is fascinating to compare, where Cube talks about headless BI and MetricFlow about the metrics layer. As discussed above, you understand why both try to address not only the metrics part but also data modeling and abstract away all data sources by adding access control, caching, and APIs. Overview of a Semantic Layer by Cube (top) and MetricFlow (bottom) When do you need such a tool? There are two initial use cases: First, if you won’t build a significant data architecture, you could start with your sources and access your data sources or cloud warehouse. The second one is when your company size and data-savvy people are growing. More people need transformations, defining metrics, or adding many data and SaaS tools; you need alignment around definitions. Some trends: Supergrain initially also had a metrics layer but pivoted to personalized marketing. If we look solely at the GitHub stars, it is a bit unfair for MetricFlow as they worked to log on that problem but only open-sourced it, but still, it gives a good impression about cube.js. They also just announced the Unbundling of Looker in combination with Superset. Furthermore, there is a dedicated Semantic Layer Summit. Open-source semantic layer solutions compared on GitHub Star History A deeper comparison of the tools you find on How is this different from. There is an extensive list of closed-source solutions such as uMetric by Uber, Minerva by Airbnb, Veezoo, and more. ℹ️ Unpopular Opinion: The Metrics Stores are just BI Tools that evolve into full-fledged BI tools over time [Tweet]Nick Hande, Co-founder of Transform, replied: “I think I actually agree with this. But, I don’t think BI Tools will exist as they do today. Instead, there will be data applications that will rely on semantics expressed in Metrics Stores. The Metrics Store will need governance + discovery that includes graphs(“BI”) + metadata”. ℹ️ An interesting announcement about the Customer Data Platform (CDP), a relatively new term, and companies already pivoting away from it. Hightouch talks about “legacy CPD”. Fascinating to me, because the new data architecture looks like a semantic layer on top of the data warehouse, or is it just me? Problems of a Semantic Layer Besides all the positive things mentioned, some challenges come with a semantic layer. First of all, it is another layer that needs integration with many other tools. Another significant downside is having to learn another solution that needs to be operationalized as a critical component of your data stack. The cost of maintaining and creating such a layer is high. There is also a hidden complexity in generating these queries on the fly. Every query needs to be generated for different SQL dialects–e.g., Postgres SQL and Oracle SQL are not 1:1 the same. That can be both problematic in terms of latency but also in terms of producing a faulty SQL query. The counter-argument is that it’s still easier to maintain copies of data sets and re-do the metrics repetitively inside a BI dashboard. When you use extensive exposed APIs, you might run into performance issues, as pulling lots of data over REST or GraphQL is less than ideal. You can always switch to SQL directly, but mostly with losing some comfort. As always, it depends on the criticality of centrally defined metrics that everyone agrees on, or you can live in minor drift away in different tools. ℹ️ What others are saying: JP Monteiro is saying in his deep dive: “I find it unlikely that the best practice will still be to have one place to define metrics and one place to define dimensions, once the querying layer part is solved. In fact, lineage between concepts is part of “semantics”: it helps us understand how one concept is related to another —which makes you really question if ‘column-level lineage’ is the right level of abstraction we should talk about: columns are too raw". What’s the Difference to OLAP, Data Cataloging, Virtualizations, or Mesh As the semantic layer is something central and intertwined with lots of related data engineering concepts, we discuss here how they relate to each other. OLAP Cubes As part of the semantic layer, the cache layer can be seen as a replacement for modern OLAP cubes such as Apache Druid, Apache Pinot, and ClickHouse. Similar attributes with defining the queries ad-hoc and delivering sub-second query times. OLAP cubes are the fastest way to query your data if you do not have updates in your data. Compared to an OLAP solution, the benefit of the semantic layer is avoiding reingesting your data into another tool and format; it happens under the hood, for good or worse. A caching layer is another complex piece, as it's always outdated as you add more data; it needs to constantly update its cache as you do not want to query old invalidated data. That's also where it gets interesting how semantic layer tools solve it. For example, Cube tried to implement with Redis but reverted and built their own caching layer from scratch. On the upside, OLAP cubes have built-in computing where you can run heavy queries. Data Virtualization and Federation Data Virtualization comes up in many discussions related to the semantic layer. Still, even more to the semantic layer is data virtualization, with tools such as Dremio that try to have all data in-memory with technologies such as Apache Arrow. Data federation, very similar to virtualization, mostly referred to technologies like Presto or Trino. These are versions of a semantic layer, including a cache layer with access management, data governance, and many more. It has a powerful option to join data in its semantic layer. In a way, they are the perfect semantic layer, although Dremio, and also Presto have branded themself as an open data lakehouse platform lately. Which brings us to another question. Is a lakehouse nothing else than a semantic layer? In a way, they have similar attributes, but a lakehouse includes an open storage layer (Delta Lake, Iceberg, Hudi). In contrast, the semantic layer is more ad-hoc, query-time driven. Still, the lakehouse from Databricks tries to store only once and avoid data movement as much as possible by querying it with their compute engine Photon. The problem is, the platform itself is not open-source, only storage, and the metrics are not defined in a declarative way. They are mostly entangled in notebooks or UI-based tools. ℹ️ Dremio has patented database-like indexes on source systems with Data Reflections. They are producing more cost-effective query plans than performing query push-downs to the data sources. Data Mesh and Data Contracts Two recent popular terms are Data Mesh and Data Contract. Both are pulling in the same direction of giving more power to the domain experts who know the data best, away from engineers who should be more focused on a stable system. In my opinion, giving more control also needs an easier way for these domain experts to define their metrics in a standardized way, being declarative directly on the Data Assets, but with a standardized tool—which is the semantic layer for me. But in contrast to data mesh, which is decentralized, the semantic layer fancies a very centralized approach for metrics. Data Catalogs A data catalog is another way of centralizing metrics and Data Products. It’s similar to a semantic layer but focuses more on the physical data assets than the metrics query. Plus, it has another focus: providing a Google search to your data assets such as dashboards, warehouse tables, or models. Modern tools such as Amundsen or DataHub allow you to rate and comment on the data assets, adding metadata such as an owner so that people easily find the best data collaboratively set for their job. The Semantic Warehouse A Semantic Warehouse is a term I heard from Chad Sanderson for the first time a month ago. He does a great job putting the words semantic layer, semantic mapping, metrics layer, and data catalog on a data map, as seen in the image below. An overview of Semantic Warehouse by Chad Sanderson What’s interesting here is that the semantic layer is directly on top of the apps, services, and DB between semantic mappings and the metrics layer. It acts as a Data Contract between the real world and the data team. In my opinion, this is what a semantic layer is described in great detail above. Analytics API In the above semantic warehouse illustration, the semantic mapping seems to be the data orchestrator, and the metrics layer is the subset of the semantic layer that holds the metrics themselves. This is interesting because I implemented such a thing (parts of it), and I wrote about it in a similar way in Building an Analytics API with GraphQL. I called it Analytics API with its core components of an API and query engine, data catalog, data orchestrator, a SQL connector, and of course, the metrics layer. It has the same function and the same component but is visualized as a single analytics API component. Queries through the access layer connecting heterogeneous data stores. Will the Semantic Layer Get More Adoption? The semantic layer is an abstract construct that is hard to grasp. It has many touching points with existing concepts and similarities with other upcoming ones. If it were a new construct, I'd say it's just another buzzword. But as it started in 1991, as seen in the history of the semantic layer, and evolved into the modern data stack and adapted to today's needs. It's still hard to implement all features we've discussed here; that's the theoretical view. But I believe if you start with getting a more diverse architecture with lots of spread-out tools, the semantic layer has its stands for staying. Start with your basic requirements. Maybe you need a central access layer for people to access data quickly. Or you do not want to add another complex layer with an OLAP cube on top and search for an efficient cache layer. Or most importantly, if you're going to define your metric in a central place with a thin layer, start with the concept of the semantic layer and its definition of metrics. Learn along the way. Check out the above tools mentioned and see if they work out. These days, the tech makes it very easy to start a POC, e.g., define some of your BI metrics inside a semantic layer and sync it into your BI tool. Try to get a feeling for it. Besides the must needs such as data integration, transformation at ingest, visualizing, and orchestrating, I see the semantic layer as the next step for defining metrics in a standardized way. — If you want to read more, I pulled together an extensive list of other articles on that topic in our data glossary on What is a Semantic Layer.

Every project is different. This is true for data streaming, analytics, and other software development. The following shows three case studies with significantly different architectures and technologies for data warehouse modernization. The examples come from various verticals: software and cloud business, financial services, logistics and transportation, and the travel and accommodation industry. Confluent: Data Warehouse Modernization From Batch ETL With Stitch to Streaming ETL With Kafka The article "Streaming ETL SFDC Data for Real-Time Customer Analytics" explores how Confluent eats its dog food to modernize the internal data warehouse pipeline. The use case is straightforward and standard across most organizations: Extract, transform, and load (ETL) Salesforce data into a Google BigQuery data warehouse, so that the business can use the data. But it is more complex than it sounds. Organizations often rely on a third-party ETL tool to periodically load data from a CRM and other applications to their data warehouse. These batch tools introduce a lag between when the business events are captured in Salesforce and when they are made available for consumption and processing. The batch workloads commonly result in discrepancies between Salesforce reports and internal dashboards, leading to concerns about the integrity and reliability of the data. Confluent used Talend's Stitch batch ETL tool in the beginning. The old architecture looked like this: The consequence of batch ETL and a 3rd party tool in the middle lead to insufficient and inconsistent information updates. Over the past few years, Confluent has invested in building stream processing capabilities into the internal data warehouse pipeline. Confluent leverages its own fully managed Confluent Cloud connectors (in this case, the Salesforce CDC source and BigQuery sink connectors), Schema Registry for data governance, and ksqlDB + Kafka Streams for reliable streaming ETL to send SFDC data to BigQuery. Here is the modernized architecture: PayPal: Reducing the Time for Readouts From 12 Hours to a Few Seconds for 30 Billion Events Per Day PayPal has plenty of Kafka projects for many critical and analytical workloads. In this use case, it scales the Kafka Consumer for 30-35 billion events per day to migrate its analytical workloads to the Google Cloud Platform (GCP). A streaming application ingests the events from Kafka directly to BigQuery. This is a critical project for PayPal as most of the analytical readouts are based on this. The outcome of the data warehouse modernization and building a cloud-native architecture: reduce the time for readouts from 12 hours to a few seconds. Read more about this success story in the PayPal technology blog. Shippeo: From On-Premise Databases to Multiple Cloud-Native Data Lakes Shippeo provides real-time and multimodal transportation visibility for logistics providers, shippers, and carriers. Its software uses automation and artificial intelligence to share real-time insights, enable better collaboration, and unlock your supply chain’s full potential. The platform can give instant access to predictive, real-time information for every delivery. Shippeo described how they integrated traditional databases (MySQL and PostgreSQL) and cloud-native data warehouses (Snowflake and BigQuery) with Apache Kafka and Debezium: This is an excellent example of cloud-native enterprise architecture leveraging a "best-of-breed" approach for data warehousing and analytics. Kafka decouples the analytical workloads from the transactional systems and handles the backpressure for slow consumers. Sykes Cottages: Fully Managed End-to-End Pipeline With Confluent Cloud, Kafka Connect, Snowflake Sykes Holiday Cottages are one of the UK's leading and fastest-growing independent holiday cottage rental agencies representing over 19,000 cottages across the UK, Ireland, and New Zealand. The experience of its customers on the web is a top priority and is one way to stay competitive. The goal is to match customers to their perfect holiday cottage experience and delight at each stage along the way. Getting the data pipeline to fuel this innovation is critical. Data warehouse modernization and data streaming enabled new ways to further innovate the web experience through a data-driven approach. From Inconsistent and Slow Batch Workloads... While serving its purpose for several years, the existing pipeline had problems impairing this cycle. Very early in this pipeline, the ETL process turned the data into rows and columns (structured data). Various copies were made, and the results were presented via a static report. Data engineers were needed for changes, such as new events or contextual information. The scale was also challenging, as this has to be done manually in the main. Critically keeping the data in a semi-structured format until it is ingested into the warehouse and then using ELT to do a single transformation of the data, Sykes Holiday Cottages can simplify the pipeline and make it much more agile. ...To Event-Based Real-Time Updates and Continuous Stream Processing New web events (and any context that goes with it) can be wrapped up within a message and can flow all the way to the warehouse without a single code change. The new events are then available to the web teams either through a query or the visualization tool. The current throughput is around 50k (peaking at over 300k) messages per minute. As new events are captured, this will grow considerably. Additionally, each of the above components must scale accordingly. The new architecture enables the web teams to capture new events and analyze the data using self-service tools with no dependency on data engineering. In conclusion, the business case for doing this is compelling. Based on our testing and projections, we expect at least 10x ROI over three years for this investment. In Sykes Holiday Cottages' blog post, learn more details: Why Sykes Cottages partnered with Snowflake and Confluent to drive enhanced customer experience. DoorDash: From Multiple Pipelines to Data Streaming for Snowflake Integration Even digital natives — that started their business in the cloud without legacy applications in their own data centers — need to modernize the enterprise architecture to improve business processes, reduce costs, and provide real-time information to its downstream applications. It is cost-inefficient to build multiple pipelines that are trying to achieve similar purposes. DoorDash used cloud-native AWS messaging and streaming systems like Amazon SQS and Amazon Kinesis for data ingestion into the Snowflake data warehouse: Mixing different kinds of data transport and going through multiple messaging/queueing systems without carefully designed observability around it leads to difficulties in operations. These issues resulted in high data latency, significant cost, and operational overhead at DoorDash. Therefore, DoorDash moved to a cloud-native streaming platform powered by Apache Kafka and Apache Flink for continuous stream processing before ingesting data into Snowflake: The move to a data streaming platform provides many benefits to DoorDash: Heterogeneous data sources and destinations, including REST APIs using the Confluent rest proxy Easily accessible End-to-end data governance with schema enforcement and schema evolution with Confluent Schema Registry Scalable, fault-tolerant, and easy to operate for a small team All the details about this cloud-native infrastructure optimization are in DoorDash's engineering blog post: "Building Scalable Real-Time Event Processing with Kafka and Flink." Real-World Case Studies for Cloud-Native Projects Prove the Business Value Data warehouse and data lake modernization only make sense if there is a business value. Elastic scale, reduced operations complexity, and faster time to market are significant advantages of cloud services like Snowflake, Databricks, or Google BigQuery. Data streaming plays a vital role in these initiatives to integrate with legacy and cloud-native data sources, continuous streaming ETL, true decoupling between the data sources, and multiple data sinks (lakes, warehouses, business applications). The case studies of Confluent, PayPal, Shippeo, Sykes Cottages, and DoorDash showed their different success stories of moving into cloud-native infrastructure to rain real-time visibility and analytics capabilities. Elastic scale and fully-managed end-to-end pipelines are crucial success factors in gaining business value with consistently up-to-date information.