Skip to content

Commit 79b5cdd

Browse files
hailin0hailin0
authored
[Feature][Connector-V2][Elasticsearch] Support https protocol (#3997)
* support https protocol * compatible with opensearch
1 parent 72097c8 commit 79b5cdd

15 files changed

Lines changed: 570 additions & 68 deletions

File tree

‎docs/en/connector-v2/sink/Elasticsearch.md‎

Lines changed: 86 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -19,18 +19,24 @@ Engine Supported
1919

2020
## Options
2121

22-
| name | type | required | default value |
23-
|-----------------|--------|----------|---------------|
24-
| hosts | array | yes | - |
25-
| index | string | yes | - |
26-
| index_type | string | no | |
27-
| primary_keys | list | no | |
28-
| key_delimiter | string | no | `_` |
29-
| username | string | no | |
30-
| password | string | no | |
31-
| max_retry_count | int | no | 3 |
32-
| max_batch_size | int | no | 10 |
33-
| common-options | | no | - |
22+
| name | type | required | default value |
23+
|-------------------------|---------|----------|---------------|
24+
| hosts | array | yes | - |
25+
| index | string | yes | - |
26+
| index_type | string | no | |
27+
| primary_keys | list | no | |
28+
| key_delimiter | string | no | `_` |
29+
| username | string | no | |
30+
| password | string | no | |
31+
| max_retry_count | int | no | 3 |
32+
| max_batch_size | int | no | 10 |
33+
| tls_verify_certificate | boolean | no | true |
34+
| tls_verify_hostnames | boolean | no | true |
35+
| tls_keystore_path | string | no | - |
36+
| tls_keystore_password | string | no | - |
37+
| tls_truststore_path | string | no | - |
38+
| tls_truststore_password | string | no | - |
39+
| common-options | | no | - |
3440

3541

3642
### hosts [array]
@@ -61,6 +67,30 @@ one bulk request max try size
6167
### max_batch_size [int]
6268
batch bulk doc max size
6369

70+
### tls_verify_certificate [boolean]
71+
72+
Enable certificates validation for HTTPS endpoints
73+
74+
### tls_verify_hostname [boolean]
75+
76+
Enable hostname validation for HTTPS endpoints
77+
78+
### tls_keystore_path [string]
79+
80+
The path to the PEM or JKS key store. This file must be readable by the operating system user running SeaTunnel.
81+
82+
### tls_keystore_password [string]
83+
84+
The key password for the key store specified
85+
86+
### tls_truststore_path [string]
87+
88+
The path to PEM or JKS trust store. This file must be readable by the operating system user running SeaTunnel.
89+
90+
### tls_truststore_password [string]
91+
92+
The key password for the trust store specified
93+
6494
### common options
6595

6696
Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details
@@ -92,6 +122,49 @@ sink {
92122
}
93123
```
94124

125+
SSL (Disable certificates validation)
126+
127+
```hocon
128+
sink {
129+
Elasticsearch {
130+
hosts = ["https://localhost:9200"]
131+
username = "elastic"
132+
password = "elasticsearch"
133+
134+
tls_verify_certificate = false
135+
}
136+
}
137+
```
138+
139+
SSL (Disable hostname validation)
140+
141+
```hocon
142+
sink {
143+
Elasticsearch {
144+
hosts = ["https://localhost:9200"]
145+
username = "elastic"
146+
password = "elasticsearch"
147+
148+
tls_verify_hostname = false
149+
}
150+
}
151+
```
152+
153+
SSL (Enable certificates validation)
154+
155+
```hocon
156+
sink {
157+
Elasticsearch {
158+
hosts = ["https://localhost:9200"]
159+
username = "elastic"
160+
password = "elasticsearch"
161+
162+
tls_keystore_path = "${your elasticsearch home}/config/certs/http.p12"
163+
tls_keystore_password = "${your password}"
164+
}
165+
}
166+
```
167+
95168
## Changelog
96169

97170
### 2.2.0-beta 2022-09-26
@@ -101,3 +174,4 @@ sink {
101174
### next version
102175

103176
- [Feature] Support CDC write DELETE/UPDATE/INSERT events ([3673](https://github.com/apache/incubator-seatunnel/pull/3673))
177+
- [Feature] Support https protocol & compatible with opensearch ([3997](https://github.com/apache/incubator-seatunnel/pull/3997))

‎docs/en/connector-v2/source/Elasticsearch.md‎

Lines changed: 93 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -19,16 +19,23 @@ support version >= 2.x and < 8.x.
1919

2020
## Options
2121

22-
| name | type | required | default value |
23-
|-------------|--------|----------|---------------|
24-
| hosts | array | yes | - |
25-
| username | string | no | - |
26-
| password | string | no | - |
27-
| index | string | yes | - |
28-
| source | array | no | - |
29-
| scroll_time | string | no | 1m |
30-
| scroll_size | int | no | 100 |
31-
| schema | | no | - |
22+
| name | type | required | default value |
23+
|-------------------------|---------|----------|---------------|
24+
| hosts | array | yes | - |
25+
| username | string | no | - |
26+
| password | string | no | - |
27+
| index | string | yes | - |
28+
| source | array | no | - |
29+
| scroll_time | string | no | 1m |
30+
| scroll_size | int | no | 100 |
31+
| schema | | no | - |
32+
| tls_verify_certificate | boolean | no | true |
33+
| tls_verify_hostnames | boolean | no | true |
34+
| tls_keystore_path | string | no | - |
35+
| tls_keystore_password | string | no | - |
36+
| tls_truststore_path | string | no | - |
37+
| tls_truststore_password | string | no | - |
38+
| common-options | | no | - |
3239

3340

3441

@@ -59,8 +66,38 @@ Maximum number of hits to be returned with each Elasticsearch scroll request.
5966
The structure of the data, including field names and field types.
6067
If you don't config schema, you must config `source`.
6168

69+
### tls_verify_certificate [boolean]
70+
71+
Enable certificates validation for HTTPS endpoints
72+
73+
### tls_verify_hostname [boolean]
74+
75+
Enable hostname validation for HTTPS endpoints
76+
77+
### tls_keystore_path [string]
78+
79+
The path to the PEM or JKS key store. This file must be readable by the operating system user running SeaTunnel.
80+
81+
### tls_keystore_password [string]
82+
83+
The key password for the key store specified
84+
85+
### tls_truststore_path [string]
86+
87+
The path to PEM or JKS trust store. This file must be readable by the operating system user running SeaTunnel.
88+
89+
### tls_truststore_password [string]
90+
91+
The key password for the trust store specified
92+
93+
### common options
94+
95+
Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details
96+
6297
## Examples
98+
6399
simple
100+
64101
```hocon
65102
Elasticsearch {
66103
hosts = ["localhost:9200"]
@@ -69,6 +106,7 @@ Elasticsearch {
69106
}
70107
```
71108
complex
109+
72110
```hocon
73111
Elasticsearch {
74112
hosts = ["elasticsearch:9200"]
@@ -94,8 +132,52 @@ Elasticsearch {
94132
}
95133
```
96134

135+
SSL (Disable certificates validation)
136+
137+
```hocon
138+
source {
139+
Elasticsearch {
140+
hosts = ["https://localhost:9200"]
141+
username = "elastic"
142+
password = "elasticsearch"
143+
144+
tls_verify_certificate = false
145+
}
146+
}
147+
```
148+
149+
SSL (Disable hostname validation)
150+
151+
```hocon
152+
source {
153+
Elasticsearch {
154+
hosts = ["https://localhost:9200"]
155+
username = "elastic"
156+
password = "elasticsearch"
157+
158+
tls_verify_hostname = false
159+
}
160+
}
161+
```
162+
163+
SSL (Enable certificates validation)
164+
165+
```hocon
166+
source {
167+
Elasticsearch {
168+
hosts = ["https://localhost:9200"]
169+
username = "elastic"
170+
password = "elasticsearch"
171+
172+
tls_keystore_path = "${your elasticsearch home}/config/certs/http.p12"
173+
tls_keystore_password = "${your password}"
174+
}
175+
}
176+
```
177+
97178
## Changelog
98179

99180
### next version
100181

101-
- Add Elasticsearch Source Connector
182+
- Add Elasticsearch Source Connector
183+
- [Feature] Support https protocol & compatible with opensearch ([3997](https://github.com/apache/incubator-seatunnel/pull/3997))

‎release-note.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
### Transformer
1111
- [Spark] Support transform-v2 for spark (#3409)
1212
- [ALL]Add FieldMapper Transform #3781
13+
### Connectors
14+
- [Elasticsearch] Support https protocol & compatible with opensearch
1315

1416
## Improves
1517
### Connectors

‎seatunnel-connectors-v2/connector-elasticsearch/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,5 +49,10 @@
4949
<artifactId>seatunnel-format-json</artifactId>
5050
<version>${project.version}</version>
5151
</dependency>
52+
<dependency>
53+
<groupId>io.airlift</groupId>
54+
<artifactId>security</artifactId>
55+
<version>206</version>
56+
</dependency>
5257
</dependencies>
5358
</project>

0 commit comments

Comments
 (0)