用JetBrains Http Client发送 elastic search bulk(多行json的http请求)
提供一种解决方法, 使用外部文件的方式, 可以在JetBrains Http Client使用es的_bulk api,或者其它此类型的http请求(包含多行json)
2023-09-02
描述
某些 http api 的 request body 包含了多行json内容, 并且在结尾需要额外的一个换行,
如elasticsearch 的_bulk api,
_bulk.sh
# curl -X POST "localhost:9200/sample_bulk_1/_bulk" -H 'Content-Type: application/json' -d'
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"John Doe","desc":"Elasticsearch is powerful","age":30}
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"Jane Smith","desc":"I love working with Elasticsearch","age":25}
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"Alice Johnson","desc":"Searching made easy with Elasticsearch","age":40}
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"Bob Brown","desc":"Elasticsearch for beginners","age":35}
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"Sarah Davis","desc":"Mastering Elasticsearch queries","age":28}
#{"index":{"_index":"sample_bulk_1"}}
#{"name":"Michael Wilson","desc":"","age":-1}
#'
这种请求在 postman 中可以直接构建并使用, 没有任何问题
但是如果习惯了在JetBrains Ide(Intellij Idea, PyCharm, WebStorm...)中使用 Http Client构建请求, 就会遇到问题
POST {{es-host}}/_bulk
Content-Type: application/json
Authorization: Basic {{es-username}} {{es-password}}
{"index":{"_index":"sample_bulk_1"}}
{"name":"John Doe","desc":"Elasticsearch is powerful","age":30}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Jane Smith","desc":"I love working with Elasticsearch","age":25}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Alice Johnson","desc":"Searching made easy with Elasticsearch","age":40}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Bob Brown","desc":"Elasticsearch for beginners","age":35}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Sarah Davis","desc":"Mastering Elasticsearch queries","age":28}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Michael Wilson","desc":"","age":-1}
#end
出现问题的原因是: _bulk需要的最后一个换行, 没有包含在请求中
通过各种关键字检索后发现issue: IJPL-67362
里面提供了一种解决方案
解决方案
使用外部文件作为json输入
< _bulk.ndjson
新的请求:
POST {{es-host}}/_bulk
Content-Type: application/json
Authorization: Basic {{es-username}} {{es-password}}
< _bulk.ndjson
_bulk.json
{"index":{"_index":"sample_bulk_1"}}
{"name":"John Doe","desc":"Elasticsearch is powerful","age":30}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Jane Smith","desc":"I love working with Elasticsearch","age":25}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Alice Johnson","desc":"Searching made easy with Elasticsearch","age":40}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Bob Brown","desc":"Elasticsearch for beginners","age":35}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Sarah Davis","desc":"Mastering Elasticsearch queries","age":28}
{"index":{"_index":"sample_bulk_1"}}
{"name":"Michael Wilson","desc":"","age":-1}
#a newline in the end 👆
其它, 为什么要使用JetBrains Http Client?
虽然postman已经很好用了, 但是postman的一些缺点让我选择使用JetBrains Http Client作为请求客户端:
- 卡顿, 闲置时莫名其妙的cpu占用
- 必须联网
- 请求不能保存为文本格式(最重要的)
- 使用文件系统进行分类和管理
- 作为git项目
JetBrains Http Client.http, 相比postman功能简陋, 支持的范围很小, 因此有了这篇文章:使用额外的补充方法来满足特别要求,
但是需求场景是无限的, 其它的一些场景无法支持, 那就只能放弃了