Set the auth_typeto oauth2 while calling the REST API to create the Pulsar source. This usually works with a local Pulsar for test purpose only. Set the auth_typeto an empty string while calling the REST API. There are 3 types of supported authentications: Go back to the "Service Accounts" page.And then click the "POLICIES" tab, and add the service account to the topic's "Authorization" list. This can be done by first, select "Topics", pick one topic (or create a new one) you want to use, click it. Assign read and/or write permissions (depends on if you want to create a source or a sink) of a topic to the service account (or you can have two service accounts, one for read, one for write).Create a service account, you don't need "Super Admin" permission.Go to the select "Service Accounts" from the navigation panel on the left side (you need to select an instance on the home page to see the navigation panel).In order to connect to StreamNative Cloud, you will need to setup a service account. Supported Pulsar Version, Deployment and Authentication īoth Apache Pulsar and StreamNative Cloud are supported. Recently Timeplus adds the first-class integration for Apache Pulsar as both a data source and a data sink. You can convert them back to DateTime via toTime function.Īpache® Pulsar™ is a cloud-native, distributed, open source messaging and streaming platform for real-time workloads. Datetime or timestamp will be saved as a String column.Values in number or boolean types in the JSON message will be converted to corresponding type in the stream.For nested attributes, the element will be saved as a String column and later you can query them with one of the JSON functions. The topic level JSON attributes will be converted to stream columns.Currently we only support JSON format for the messages in Kafka topics.Please make sure Timeplus can reach out to your Kafka broker(s) Notes for Kafka source You may not need to specify the SASL Plain as the authentication method. Similar steps as loading data from Confluent Cloud. Finish the rest of the wizard and your streaming data will be available in the new stream immediately.Click Next to preview the streaming data from the specified Kafka source and choose a column as the event time.Alternatively, you can disable the stream creation and choose an existing stream from the list. By default, the source can create a new stream in Timeplus.When AVRO is selected, you also need to specify the address, API key, and secret key for the schema registry. This would be more convenient for you to query, but won't support schema evolution. If you turn it on, then the top level attribute in the AVRO message will be put into different columns. By default it's turned off, meaning the entire message will be saved as a string. If you choose AVRO, there is an option for 'Auto Extraction'.If the data in the Kafka topic is in JSON format, but the schema may change over time, we recommend you to choose the Text format, so that the entire JSON document will be saved as a string, and you can apply JSON related functions to extract value, even the schema is changed. For Data Format, we currently support JSON, AVRO and Text format.Type in the API Key and Secret Key for your cluster. When you select Confluent Cloud, the SASL Plain is automatically selected.Specify the broker(s) URL, such as .cloud:9092.Choose Confluent Cloud for the deployment type of the Kafka.Specify a name for this data source, and optionally provide a readable description.Then click Kafka and click the Start button Click the Add Data from the navigation menu.You can also create external streams to analyze data in Confluent/Kafka/Redpanda without moving data. With our strong partnership with Confluent, you can load your real-time data from Confluent Cloud, Confluent Platform, or Apache Kafka into the Timeplus streaming engine. Add new sources via web console Load streaming data from Apache Kafka Īs of today, Kafka is the primary data source (and sink) for Timeplus. Timeplus supports a wide range of data sources.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |