972-836-7750 Request a Demo Talk to an EDI/API Expert Search
    See all results Browse the sitemap Tip: press / to search from any page
    Talk to an EDI/API Expert Request a Demo Call Sales 972-836-7750

    Messaging and events

    Apache Kafka and Confluent integration

    Apache Kafka is an event streaming platform: a durable log of events that many applications read at their own pace. Confluent offers Kafka as a managed cloud service and a platform with schema management and connectors. ECOM streams partner and EDI data into Kafka topics and turns events into partner documents.

    Project
    Apache Kafka (open source)
    Commercial platform
    Confluent Platform and Confluent Cloud
    Model
    Topics, partitions, consumer groups
    Data contracts
    Schema Registry (Avro, JSON Schema, Protobuf)
    Talk to an EDI/API Expert
    Where Apache Kafka/Confluent fitsA hub diagram: order services, the warehouse system, stores and e-commerce, and devices publish Kafka events; the ECOM B2B gateway reads topics with schemas and maps events for the ERP, warehouse, EDI partners and APIs.Where Apache Kafka/Confluent fitsPartners, the ECOM gateway and your systemsEVENT PRODUCERSOrder servicesWarehouse (WMS)Stores and e-commerceIoT and devicesDESTINATIONSERPWarehouse (WMS)EDI partnersAPIs and appsECOM B2B gatewayApache Kafka/ConfluentTopics and partitionsSchemasMap events
    Where Apache Kafka/Confluent fitsA hub diagram: order services, the warehouse system, stores and e-commerce, and devices publish Kafka events; the ECOM B2B gateway reads topics with schemas and maps events for the ERP, warehouse, EDI partners and APIs.Where Apache Kafka/ConfluentfitsPartners, the ECOM gateway and your systemsEVENT PRODUCERSOrder servicesWarehouse (WMS)Stores and e-commerceIoT and devicesECOM B2B gatewayApache Kafka/ConfluentTopics andpartitionsSchemasMap eventsDESTINATIONSERPWarehouse (WMS)EDI partnersAPIs and apps

    Events flow through Kafka topics; the gateway turns them into partner documents and back.

    Expand diagram Download SVG

    Overview

    What Apache Kafka/Confluent is

    Kafka stores events in topics. Each topic is split into partitions, and each event in a partition has an offset, its position in the log. Consumers read events and record the offset they have reached, so they can stop and resume, and several independent applications can read the same events.

    Events are kept for a configured retention period, not deleted when read. That makes Kafka a source of truth for streams such as orders, shipments and inventory changes. Schemas in a schema registry keep producers and consumers agreed on the structure of each event.

    Where businesses use it

    • Order and inventory streams

      Every order, allocation and stock change published once and used by many systems.

    • Data platforms

      Feeding analytics, data lakes and dashboards in near real time.

    • Microservices

      Services communicating through events instead of direct calls.

    How it works

    How Kafka streaming works

    A producer sends an event with a key and a value to a topic. The key decides the partition, so events for the same order stay in order. Consumers in a consumer group share the partitions between them; each partition is read by one member of the group.

    Consumers commit offsets after processing. If a consumer fails, another takes over from the last committed offset. Idempotent producers and transactions avoid duplicates where exactly-once processing is needed.

    Partner orders published to KafkaA sequence diagram: the ECOM gateway translates a partner order and produces an event keyed by order number to an orders topic, an ERP consumer reads it, processes it and commits its offset, and an analytics consumer reads the same event independently.Partner orders published to KafkaProduce, consume, commit offsetECOM gatewayKafka topicERP consumerAnalyticsProduce order event (key = order)Poll eventsCreate orderCommit offsetRead same event independently
    Partner orders published to KafkaA sequence diagram: the ECOM gateway translates a partner order and produces an event keyed by order number to an orders topic, an ERP consumer reads it, processes it and commits its offset, and an analytics consumer reads the same event independently.Partner orders published toKafkaProduce, consume, commit offsetECOMgatewayKafka topicERPconsumerAnalyticsProduce order event (key =order)Poll eventsCreate orderCommit offsetRead same eventindependently

    One event serves several consumers. Each one tracks its own offset, so they never interfere.

    Expand diagram Download SVG

    In depth

    Kafka integration in practice

    • Keys and ordering

      Order is kept within a partition, so keys are chosen to keep related events together.

    • Schemas

      Avro, JSON Schema or Protobuf schemas in a registry are used to validate and evolve events.

    • Delivery guarantees

      At-least-once is common; idempotent producers and transactions are used where duplicates must be avoided.

    • Retention and replay

      Events can be replayed from an earlier offset to rebuild or reprocess data.

    • Security

      TLS, SASL authentication and access control lists protect clusters and topics.

    • Dead-letter topics

      Events that cannot be processed go to a separate topic for review.

    In practice

    Example: inventory events to retailer EDI

    Inventory changes are published to Kafka. Retailers need a daily X12 846.

    Example: inventory events to retailer EDIFive steps: consume inventory events from Kafka, keep the latest quantity per item and location, build each retailer's X12 846, send it, and commit offsets.Example: inventory events to retailer EDIA practical example, step by step1ConsumechangesThe gateway readsinventory eventsfrom the topic.Step 12Keep the latestThe latest quantityper item andlocation is kept.Step 23Build the 846Each retailer'sinventory advice ismapped.Step 34SendThe 846 goes toeach retailer.Step 45CommitOffsets arecommitted aftersending.Step 5
    Example: inventory events to retailer EDIFive steps: consume inventory events from Kafka, keep the latest quantity per item and location, build each retailer's X12 846, send it, and commit offsets.Example: inventory events toretailer EDIA practical example, step by step1Consume changesThe gateway reads inventory eventsfrom the topic.Step 12Keep the latestThe latest quantity per item and locationis kept.Step 23Build the 846Each retailer's inventory advice ismapped.Step 34SendThe 846 goes to each retailer.Step 45CommitOffsets are committed after sending.Step 5

    The stream keeps every change; the 846 gives each retailer the current picture.

    Expand diagram Download SVG

    Connections

    How Apache Kafka/Confluent connects your business

    • Trading partners

      EDI documents from partners become Kafka messages for internal systems, and events become partner documents.

    • ERP and business systems

      Orders, shipments, inventory and invoices move between applications as events, without point-to-point links.

    • APIs and files

      The same data can be published to APIs or written to files for systems that do not read the platform.

    • Business workflows

      Order-to-cash and fulfillment steps react to events as they happen.

    ECOM

    How ECOM supports Apache Kafka/Confluent

    ECOM supports Apache Kafka/Confluent with dedicated implementation, mapping, communication and support teams, and with specialists who know the industries your partners work in.

    • Implementation team

      Plans the Apache Kafka/Confluent setup with you, configures it on the gateway and runs the tests with your partners and systems.

    • Mapping team

      Builds and maintains the maps, code conversions and validation rules between partner data and your applications.

    • Communication team

      Sets up and monitors partner connections, certificates, keys and credentials, and coordinates changes with each partner.

    • Support team

      Watches the flows, investigates failed or missing documents, and works the issue through with you and your partner.

    For Apache Kafka/Confluent in particular

    • Kafka and Confluent producers and consumers with schema registry integration.
    • Topic, key, partition and security design with your platform team.
    • Mapping between Kafka messages and EDI, XML, JSON, files and your ERP, with validation.
    • Monitoring of queues, subscriptions and failed messages, with alerts and a clear path to reprocess.

    Frequently asked questions

    Is Kafka a message queue?

    Not exactly. Kafka is a durable log: events stay for a retention period and many consumers can read them independently, unlike a queue where a message is removed once consumed.

    What is the difference between Kafka and Confluent?

    Apache Kafka is the open-source project. Confluent is a company that offers Kafka as a managed cloud service and a platform with additional tools such as schema management and connectors.

    Can EDI documents be published to Kafka?

    Yes. Partner documents are translated into events with agreed schemas, and events can be turned into partner documents.

    How are duplicates avoided?

    With idempotent producers, transactions where needed, and keys that let consumers detect repeats.

    Related pages

    Talk to an EDI/API Expert

    Tell us which partners, systems and Apache Kafka/Confluent requirements you are working with. An ECOM specialist will review them with you and outline the setup, the maps and the tests.

    Talk to an EDI/API Expert Request a Demo