Build Bulk Request(Multi-line JSON HTTP Requests) Of Elastic Search with JetBrains Http Client
Provides a solution using external json file to use the es _bulk API or other similar HTTP requests (multi-line JSON) in JetBrains Http Client.
Description
Some HTTP APIs have request bodies containing multi-line JSON content, and an extra newline is needed at the end.
For example, the _bulk API of elasticsearch requires this format:
# curl -X POST "localhost:9200/sample_bulk_1/_bulk" -H 'Content-Type: application/json' -d'
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"John Doe","desc":"Elasticsearch is powerful","age":30}
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"Jane Smith","desc":"I love working with Elasticsearch","age":25}
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"Alice Johnson","desc":"Searching made easy with Elasticsearch","age":40}
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"Bob Brown","desc":"Elasticsearch for beginners","age":35}
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"Sarah Davis","desc":"Mastering Elasticsearch queries","age":28}
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"Michael Wilson","desc":"","age":-1}
#'
This request can be out-of-boxed used in postman without any issues.
However, if you are want to building requests using the Http Client in JetBrains IDE (Intellij Idea, PyCharm, WebStorm, etc.), you will encounter problems.
POST {{es-host}}/_bulk
Content-Type: application/json
Authorization: Basic {{es-username}} {{es-password}}
{"index":{"_index":"sample_bulk_1"}}
{"name":"John Doe","desc":"Elasticsearch is powerful","age":30}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Jane Smith","desc":"I love working with Elasticsearch","age":25}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Alice Johnson","desc":"Searching made easy with Elasticsearch","age":40}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Bob Brown","desc":"Elasticsearch for beginners","age":35}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Sarah Davis","desc":"Mastering Elasticsearch queries","age":28}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Michael Wilson","desc":"","age":-1}
#end
The problem occurs because the final newline required by _bulk is not included in the request body.
After searching with various keywords, I found the issue: IJPL-67362.
It provides a solution.
Solution
Use an external file as JSON input:
< _bulk.ndjson
The new request becomes:
POST {{es-host}}/_bulk
Content-Type: application/json
Authorization: Basic {{es-username}} {{es-password}}
< _bulk.ndjson
{"index":{"_index":"sample_bulk_1"}}
{"name":"John Doe","desc":"Elasticsearch is powerful","age":30}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Jane Smith","desc":"I love working with Elasticsearch","age":25}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Alice Johnson","desc":"Searching made easy with Elasticsearch","age":40}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Bob Brown","desc":"Elasticsearch for beginners","age":35}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Sarah Davis","desc":"Mastering Elasticsearch queries","age":28}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Michael Wilson","desc":"","age":-1}
#a newline in the end 👆
Furthermore, why use JetBrains Http Client?
Although postman is already very useful, some things make me to choose JetBrains Http Client as my request client:
- Lagging, unknown CPU usage when idle
- online required
- Requests cannot be saved in text format (most importantly)
- cannot use the file system for classification and management
- cannot be as a git project
The JetBrains Http Client .http has simpler functionality compared to postman, and the scope of supported features is limited, hence this article: using additional supplementary methods to meet special requirements. However, the demand scenarios are limitless, and for other scenarios that cannot be supported, one will have to give up.