Skip to content

Commit 296a8fa

Browse files
CosmosNiCosmosNi
authored
[Feature][Connector-V2][Elasticsearch] Add Runtime Fields support (Elasticsearch 7.11+) (#10201)
1 parent eeb4c0b commit 296a8fa

14 files changed

Lines changed: 696 additions & 60 deletions

File tree

‎docs/en/connectors/sink/Easysearch.md‎

Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -45,25 +45,25 @@ Engine Supported
4545

4646
## Sink Options
4747

48-
| name | type | required | default value |
49-
|-------------------------|---------|----------|---------------|
50-
| hosts | array | yes | - |
51-
| index | string | yes | - |
52-
| primary_keys | list | no | |
53-
| key_delimiter | string | no | `_` |
54-
| username | string | no | |
55-
| password | string | no | |
56-
| max_retry_count | int | no | 3 |
57-
| max_batch_size | int | no | 10 |
58-
| tls_verify_certificate | boolean | no | true |
59-
| tls_verify_hostnames | boolean | no | true |
60-
| tls_keystore_path | string | no | - |
61-
| tls_keystore_password | string | no | - |
62-
| tls_truststore_path | string | no | - |
48+
| name | type | required | default value |
49+
|------------------------|---------|----------|---------------|
50+
| hosts | array | yes | - |
51+
| index | string | yes | - |
52+
| primary_keys | list | no | |
53+
| key_delimiter | string | no | `_` |
54+
| username | string | no | |
55+
| password | string | no | |
56+
| max_retry_count | int | no | 3 |
57+
| max_batch_size | int | no | 10 |
58+
| tls_verify_certificate | boolean | no | true |
59+
| tls_verify_hostname | boolean | no | true |
60+
| tls_keystore_path | string | no | - |
61+
| tls_keystore_password | string | no | - |
62+
| tls_truststore_path | string | no | - |
6363
| tls_truststore_password | string | no | - |
64-
| schema_save_mode | enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST |
65-
| data_save_mode | enum | no | APPEND_DATA |
66-
| common-options | | no | - |
64+
| schema_save_mode | enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST |
65+
| data_save_mode | enum | no | APPEND_DATA |
66+
| common-options | | no | - |
6767

6868
### hosts [array]
6969

‎docs/en/connectors/sink/Elasticsearch.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ Engine Supported
3939
| max_retry_count | int | no | 3 |
4040
| max_batch_size | int | no | 10 |
4141
| tls_verify_certificate | boolean | no | true |
42-
| tls_verify_hostnames | boolean | no | true |
42+
| tls_verify_hostname | boolean | no | true |
4343
| tls_keystore_path | string | no | - |
4444
| tls_keystore_password | string | no | - |
4545
| tls_truststore_path | string | no | - |

‎docs/en/connectors/source/Elasticsearch.md‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ support version >= 2.x and <= 8.x.
4848
| tls_truststore_password | string | no | - |
4949
| pit_keep_alive | long | no | 60000 (1 minute) |
5050
| pit_batch_size | int | no | 100 |
51+
| runtime_fields | array | no | - |
5152
| common-options | | no | - |
5253

5354

@@ -211,6 +212,50 @@ The amount of time (in milliseconds) for which the PIT should be keep alive
211212
### pit_batch_size [int]
212213
Maximum number of hits to be returned with each PIT search request
213214

215+
### runtime_fields [array]
216+
217+
Runtime fields to be computed at query time (Elasticsearch 7.11+). Each runtime field should contain:
218+
- **name**: The name of the runtime field
219+
- **type**: The data type (boolean, date, double, geo_point, ip, keyword, long)
220+
- **script**: Painless script to compute the field value
221+
- **script_lang** (optional): Script language (default: painless)
222+
- **script_params** (optional): Script parameters
223+
224+
Example:
225+
```hocon
226+
runtime_fields = [
227+
{
228+
name = "day_of_week"
229+
type = "keyword"
230+
script = "emit(doc['timestamp'].value.dayOfWeekEnum.toString())"
231+
},
232+
{
233+
name = "total_price"
234+
type = "double"
235+
script = "emit(doc['quantity'].value * doc['price'].value)"
236+
}
237+
]
238+
```
239+
240+
**Runtime Fields Use Cases:**
241+
242+
1. **Date Extraction**: Extract day of week, month, year from timestamps
243+
2. **Calculations**: Compute derived values like total price, tax amount
244+
3. **String Operations**: Concatenate fields, extract substrings
245+
4. **Conditional Logic**: Categorize data based on conditions
246+
5. **Data Transformation**: Convert units, format values on-the-fly
247+
248+
**Performance Considerations:**
249+
- Runtime fields are computed at query time, which may impact performance for large datasets
250+
- Best suited for ad-hoc analysis, prototyping, and infrequent queries
251+
- Keep scripts simple to minimize performance impact
252+
- Consider indexing frequently used computed fields
253+
254+
**Limitations:**
255+
- Requires Elasticsearch 7.11 or higher
256+
- Only Painless scripts are supported
257+
- May be slower than indexed fields for large-scale queries
258+
214259
### common options
215260

216261
Source plugin common parameters, please refer to [Source Common Options](../common-options/source-common-options.md) for details
@@ -386,6 +431,94 @@ source {
386431
}
387432
```
388433

434+
Demo 8: Runtime Fields (Elasticsearch 7.11+)
435+
436+
> This example demonstrates how to use runtime fields to compute values at query time without reindexing data.
437+
438+
```hocon
439+
source {
440+
Elasticsearch {
441+
hosts = ["https://elasticsearch:9200"]
442+
username = "elastic"
443+
password = "elasticsearch"
444+
tls_verify_certificate = false
445+
tls_verify_hostname = false
446+
447+
index = "sales_data"
448+
449+
# Define runtime fields for dynamic computation
450+
runtime_fields = [
451+
{
452+
# Calculate total amount
453+
name = "total_amount"
454+
type = "double"
455+
script = "emit(doc['quantity'].value * doc['price'].value)"
456+
},
457+
{
458+
# Extract day of week from timestamp
459+
name = "day_of_week"
460+
type = "keyword"
461+
script = "emit(doc['order_date'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
462+
},
463+
{
464+
# Categorize orders
465+
name = "order_category"
466+
type = "keyword"
467+
script = """
468+
double amount = doc['quantity'].value * doc['price'].value;
469+
if (amount > 1000) {
470+
emit('high_value');
471+
} else if (amount > 100) {
472+
emit('medium_value');
473+
} else {
474+
emit('low_value');
475+
}
476+
"""
477+
},
478+
{
479+
# Calculate with parameters
480+
name = "price_with_tax"
481+
type = "double"
482+
script = "emit(doc['price'].value * (1 + params.tax_rate))"
483+
script_params = {
484+
tax_rate = 0.13
485+
}
486+
}
487+
]
488+
489+
# Include runtime fields in the output
490+
source = [
491+
"product_id",
492+
"quantity",
493+
"price",
494+
"order_date",
495+
"total_amount",
496+
"day_of_week",
497+
"order_category",
498+
"price_with_tax"
499+
]
500+
501+
schema = {
502+
fields {
503+
product_id = string
504+
quantity = int
505+
price = double
506+
order_date = timestamp
507+
total_amount = double
508+
day_of_week = string
509+
order_category = string
510+
price_with_tax = double
511+
}
512+
}
513+
}
514+
}
515+
516+
sink {
517+
Console {
518+
}
519+
}
520+
```
521+
389522
## Changelog
390523

391524
<ChangeLog />

‎docs/zh/connectors/sink/Easysearch.md‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -46,24 +46,24 @@ import ChangeLog from '../changelog/connector-easysearch.md';
4646
## 接收器选项
4747

4848
| 名称 | 类型 | 必需 | 默认值 |
49-
|-------------------------|---------|----|---------------|
50-
| hosts | array | 是 | - |
51-
| index | string | 是 | - |
52-
| primary_keys | list | 否 | |
53-
| key_delimiter | string | 否 | `_` |
54-
| username | string | 否 | |
55-
| password | string | 否 | |
56-
| max_retry_count | int | 否 | 3 |
57-
| max_batch_size | int | 否 | 10 |
58-
| tls_verify_certificate | boolean | 否 | true |
59-
| tls_verify_hostnames | boolean | 否 | true |
60-
| tls_keystore_path | string | 否 | - |
61-
| tls_keystore_password | string | 否 | - |
62-
| tls_truststore_path | string | 否 | - |
49+
|------------------------|---------|----|---------------|
50+
| hosts | array | 是 | - |
51+
| index | string | 是 | - |
52+
| primary_keys | list | 否 | |
53+
| key_delimiter | string | 否 | `_` |
54+
| username | string | 否 | |
55+
| password | string | 否 | |
56+
| max_retry_count | int | 否 | 3 |
57+
| max_batch_size | int | 否 | 10 |
58+
| tls_verify_certificate | boolean | 否 | true |
59+
| tls_verify_hostname | boolean | 否 | true |
60+
| tls_keystore_path | string | 否 | - |
61+
| tls_keystore_password | string | 否 | - |
62+
| tls_truststore_path | string | 否 | - |
6363
| tls_truststore_password | string | 否 | - |
64-
| schema_save_mode | enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST |
65-
| data_save_mode | enum | 否 | APPEND_DATA |
66-
| common-options | | 否 | - |
64+
| schema_save_mode | enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST |
65+
| data_save_mode | enum | 否 | APPEND_DATA |
66+
| common-options | | 否 | - |
6767

6868
### hosts [array]
6969

‎docs/zh/connectors/sink/Elasticsearch.md‎

Lines changed: 21 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -21,28 +21,28 @@ import ChangeLog from '../changelog/connector-elasticsearch.md';
2121

2222
## 选项
2323

24-
| 名称 | 类型 | 是否必须 | 默认值 |
25-
|-------------------------|---------|------|------------------------------|
26-
| hosts | array | 是 | - |
27-
| index | string | 是 | - |
28-
| schema_save_mode | string | 是 | CREATE_SCHEMA_WHEN_NOT_EXIST |
29-
| data_save_mode | string | 是 | APPEND_DATA |
30-
| index_type | string | 否 | |
31-
| primary_keys | list | 否 | |
32-
| key_delimiter | string | 否 | `_` |
33-
| username | string | 否 | |
34-
| password | string | 否 | |
35-
| max_retry_count | int | 否 | 3 |
36-
| max_batch_size | int | 否 | 10 |
37-
| tls_verify_certificate | boolean | 否 | true |
38-
| tls_verify_hostnames | boolean | 否 | true |
39-
| tls_keystore_path | string | 否 | - |
40-
| tls_keystore_password | string | 否 | - |
41-
| tls_truststore_path | string | 否 | - |
24+
| 名称 | 类型 | 是否必须 | 默认值 |
25+
|------------------------|---------|------|------------------------------|
26+
| hosts | array | 是 | - |
27+
| index | string | 是 | - |
28+
| schema_save_mode | string | 是 | CREATE_SCHEMA_WHEN_NOT_EXIST |
29+
| data_save_mode | string | 是 | APPEND_DATA |
30+
| index_type | string | 否 | |
31+
| primary_keys | list | 否 | |
32+
| key_delimiter | string | 否 | `_` |
33+
| username | string | 否 | |
34+
| password | string | 否 | |
35+
| max_retry_count | int | 否 | 3 |
36+
| max_batch_size | int | 否 | 10 |
37+
| tls_verify_certificate | boolean | 否 | true |
38+
| tls_verify_hostname | boolean | 否 | true |
39+
| tls_keystore_path | string | 否 | - |
40+
| tls_keystore_password | string | 否 | - |
41+
| tls_truststore_path | string | 否 | - |
4242
| tls_truststore_password | string | 否 | - |
43-
| common-options | | 否 | - |
44-
| vectorization_fields | array | 否 | - |
45-
| vector_dimensions | int | 否 | - |
43+
| common-options | | 否 | - |
44+
| vectorization_fields | array | 否 | - |
45+
| vector_dimensions | int | 否 | - |
4646

4747
### hosts [array]
4848

0 commit comments

Comments
 (0)