1. Environment Preparation
Download the Doris connector. The specific connectors to download are configured in config/plugin_config. Add the following connector: connector-http-base, and then use <span>sh bin/install-plugin.sh</span> to install it. If installation fails, you can go to the Central Repository: org/apache/seatunnel to download it. Place the downloaded jar file in the connectors/ directory.
2. Read Example Analysis
env { parallelism = 1 job.mode = "BATCH"}source { Http { result_table_name = "https" url = "https://jsonplaceholder.typicode.com/posts" method = "GET" format = "json" schema = { fields { userId = "bigint" id = "bigint" title = string body = string } } }}
sink { Console { }}| Attribute | Description |
| result_table_name | Defines the table name |
| url | URL of the data source |
| method | HTTP method |
| format | Return format |
| schema | Defines the fields returned |
Note that parallelism must be set to 1.
3. Writing HTTP Data Analysis
env { parallelism = 1 job.mode = "BATCH"}source { Http { result_table_name = "https" url = "https://jsonplaceholder.typicode.com/posts" method = "GET" format = "json" schema = { fields { userId = "bigint" id = "bigint" title = string body = string } } }}
sink { Http { url = "http://localhost/test/posts" }}The URL provided is for external access.If you are interested in data warehouse technologies, feel free to connect with the author for discussions.