tonglin0325的个人主页

Elasticsearch学习笔记——安装、数据导入和查询

tonglin0325

到elasticsearch网站下载最新版本的elasticsearch 6.2.1

1
2
https://www.elastic.co/downloads/elasticsearch

其他版本

1
2
https://www.elastic.co/cn/downloads/past-releases/elasticsearch-6-4-2

嫌弃官方下载速度慢的可以去华为的镜像站去

1
2
https://mirrors.huaweicloud.com/elasticsearch/6.4.2/

中文文档请参考

1
2
https://www.elastic.co/guide/cn/elasticsearch/guide/current/index.html

英文文档及其Java API使用方法请参考，官方文档比任何博客都可信

1
2
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index.html

Python API使用方法

1
2
http://elasticsearch-py.readthedocs.io/en/master/

下载tar包，然后解压到/usr/local目录下，修改一下用户和组之后可以使用非root用户启动，启动命令

1
2
./bin/elasticsearch

然后访问http://127.0.0.1:9200/

如果需要让外网访问Elasticsearch的9200端口的话，需要将es的host绑定到外网

修改 /configs/elasticsearch.yml文件，添加如下

1
2
3
network.host: 0.0.0.0
http.port: 9200

然后重启，如果遇到下面问题的话

1
2
3
4
5
[2018-01-28T23:51:35,204][INFO ][o.e.b.BootstrapChecks    ] [qR5cyzh] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

解决方法

第一个ERROR,

在文件中添加 sudo vim /etc/security/limits.conf,然后重新登录

1
2
3
4
5
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

如果你是用supervisor启动es的话,需要修改文件 vim /etc/supervisor/supervisord.conf,然后重启supervisor

1
2
3
4
[supervisord]

minfds=65536

 

第二个ERROR,在root用户下执行

临时解决

1
2
sysctl -w vm.max_map_count=262144

永久解决

1
2
3
cat /proc/sys/vm/max_map_count
sudo vim /etc/sysctl.conf

添加

1
2
vm.max_map_count=262144

然后使其生效

1
2
sysctl -p

 

接下来导入json格式的数据，数据内容如下

1
2
3
4
5
{"index":{"_id":"1"}}
{"title":"许宝江","url":"7254863","chineseName":"许宝江","sex":"男","occupation":" 滦县农业局局长","nationality":"中国"}
{"index":{"_id":"2"}}
{"title":"鲍志成","url":"2074015","chineseName":"鲍志成","occupation":"医师","nationality":"中国","birthDate":"1901年","deathDate":"1973年","graduatedFrom":"香港大学"}

 需要注意的是{“index”:{“_id”:”1”}}和文件末尾另起一行换行是不可少的

其中的id可以从0开始，甚至是abc等等

否则会出现400状态，错误提示分别为

1
2
Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_STRING]

1
2
The bulk request must be terminated by a newline [\n]"

使用下面命令来导入json文件

其中的people.json为文件的路径，可以是/home/common/下载/xxx.json

其中的es是index，people是type，在elasticsearch中的index和type可以理解成关系数据库中的database和table，两者都是必不可少的

1
2
curl -H "Content-Type: application/json" -XPOST 'localhost:9200/es/people/_bulk?pretty&refresh' --data-binary "@people.json"

 成功后的返回值是200，比如

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
{
  "took" : 233,
  "errors" : false,
  "items" : [
    {
      "index" : {
        "_index" : "es",
        "_type" : "people",
        "_id" : "1",
        "_version" : 1,
        "result" : "created",
        "forced_refresh" : true,
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 0,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "es",
        "_type" : "people",
        "_id" : "2",
        "_version" : 1,
        "result" : "created",
        "forced_refresh" : true,
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 0,
        "_primary_term" : 1,
        "status" : 201
      }
    }
  ]
}

<0>查看字段的mapping

1
2
http://localhost:9200/es/people/_mapping

 

 

 接下来可以使用对应的查询语句对数据进行查询

 <1>按id来查询

1
2
http://localhost:9200/es/people/1

 

<2>简单的匹配查询，查询某个字段中包含某个关键字的数据（GET）

1
2
http://localhost:9200/es/people/_search?q=_id:1

1
2
http://localhost:9200/es/people/_search?q=title:许

 

<3>多字段查询，在多个字段中查询包含某个关键字的数据（POST）

可以使用Firefox中的RESTer插件来构造一个POST请求，在升级到Firefox quantum之后，原来使用的Poster插件挂了

在title和sex字段中查询包含 许 字的数据

1
2
3
4
5
6
7
8
9
{
    "query": {
        "multi_match" : {
            "query" : "许",
            "fields": ["title", "sex"]
        }
    }
}

 

还可以额外指定返回值

size指定返回的数量

from指定返回的id起始值

_source指定返回的字段

highlight指定语法高亮

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
    "query": {
        "multi_match" : {
            "query" : "中国",
            "fields": ["nationality", "sex"]
        }
    },
    "size": 2,
    "from": 0,
    "_source": [ "title", "sex", "nationality" ],
    "highlight": {
        "fields" : {
            "title" : {}
        }
    }
}

<4>Boosting

用于提升字段的权重，可以将max_score的分数乘以一个系数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
    "query": {
        "multi_match" : {
            "query" : "中国",
            "fields": ["nationality^3", "sex"]
        }
    },
    "size": 2,
    "from": 0,
    "_source": [ "title", "sex", "nationality" ],
    "highlight": {
        "fields" : {
            "title" : {}
        }
    }
}

 

<5>组合查询，可以实现一些比较复杂的查询

AND -> must

NOT -> must not

OR -> should

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
    "query": {
        "bool": {
            "must": {
                "bool" : { 
                    "should": [
                      { "match": { "title": "鲍" }},
                      { "match": { "title": "许" }} ],
                    "must": { "match": {"nationality": "中国" }}
                }
            },
            "must_not": { "match": {"sex": "女" }}
        }
    }
}

 <6>模糊（Fuzzy）查询（POST）

1
2
3
4
5
6
7
8
9
10
11
12
{
    "query": {
        "multi_match" : {
            "query" : "厂长",
            "fields": ["title", "sex","occupation"],
            "fuzziness": "AUTO"
        }
    },
    "_source": ["title", "sex", "occupation"],
    "size": 1
}

 通过模糊匹配将 厂长 和 局长 匹配上

AUTO的时候，当query的长度大于5的时候，模糊值指定为2

<7>通配符（Wildcard）查询（POST）

？ 匹配任何字符

* 匹配零个或多个字

1
2
3
4
5
6
7
8
9
10
{
    "query": {
        "wildcard" : {
            "title" : "*宝"
        }
    },
    "_source": ["title", "sex", "occupation"],
    "size": 1
}

 <8>正则（Regexp）查询（POST）

1
2
3
4
5
6
7
8
9
10
{
    "query": {
        "regexp" : {
            "authors" : "t[a-z]*y"
        }
    },
    "_source": ["title", "sex", "occupation"],
    "size": 3
}

<9>短语匹配（Match Phrase）查询（POST）

短语匹配查询 要求在请求字符串中的所有查询项必须都在文档中存在，文中顺序也得和请求字符串一致，且彼此相连。

默认情况下，查询项之间必须紧密相连，但可以设置 slop 值来指定查询项之间可以分隔多远的距离，结果仍将被当作一次成功的匹配。

1
2
3
4
5
6
7
8
9
10
11
12
{
    "query": {
        "multi_match" : {
            "query" : "许长江",
            "fields": ["title", "sex","occupation"],
            "type": "phrase"
        }
    },
    "_source": ["title", "sex", "occupation"],
    "size": 3
}

 注意使用slop的时候距离是累加的，滦农局 和 滦县农业局 差了2个距离

1
2
3
4
5
6
7
8
9
10
11
12
13
{
    "query": {
        "multi_match" : {
            "query" : "滦农局",
            "fields": ["title", "sex","occupation"],
            "type": "phrase",
            "slop":2
        }
    },
    "_source": ["title", "sex", "occupation"],
    "size": 3
}

<10>短语前缀（Match Phrase Prefix）查询

1
2
https://www.elastic.co/guide/cn/elasticsearch/guide/current/prefix-query.html

比如

1
2
3
4
5
6
7
8
9
GET /my_index/address/_search
{
    "query": {
        "prefix": {
            "postcode": "W1"
        }
    }
}

　　 

一些比较复杂的DSL

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
GET index_*/_search
{
  "query": {
        "bool": {
            "must": [{
                "range" : {
                  "publish_date" : {
                      "gt" : "2014-01-01",
                      "lt" : "2019-01-07"
                  }
                }
            },
            { "multi_match": {
              "query": "免费",
              "fields":["name1","name2","name3","name4","name5","name6"]
              }
            },
            { "multi_match": {
              "query": "英语",
              "fields":["name1","name2","name3","name4","name5","name6"]
              }
            }
            
            ],
            "must_not": { "match": {"tags": "" }},
            "filter": {
                "range": { "count": { "gte": "30" ,"lte": "1000"}} 
            }
        }
    },
  "aggs": {
    "by_tags": {
      "terms": { "field": "field1"
      },
      "aggs": {
      "sales": {
         "date_histogram": {
            "field": "date",
            "interval": "day", 
            "format": "yyyy-MM-dd" 
         }
      }
   }
    }
  },
  "_source": [],
  "size": 1
}

带有去重的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
GET xxxx_2019-09-10/_search
{
  "query": {
        "bool": {
            "must": [
              {
                "range" : {
                  "xxxx" : {
                      "gt" : "2014-01-01",
                      "lt" : "2019-01-07"
                  }
                }
              },
              { 
                "terms": {
                  "xxxx": ["xxx","xxx"]
                }
              },
              { 
                "terms": {
                  "xxx": ["xxx","xxx"]
                }
              },
              { 
                "terms": {
                  "xxx": ["xxx"]
                }
              },
              {
                "bool": {
                  "should": [
                      {
                          "range": { 
                            "xxx": { "gte": 1 ,"lte": 2.99 }}
                      },
                      {"range": { 
                        "xxx": { "gte": 3.99 ,"lte": 7.99 }}
                      }
              ]}},{
                "bool": {
                  "should": [
                      {
                          "range": { 
                            "xxx": { "gte": 0 ,"lte": 100 }}
                      },
                      {"range": { 
                        "xxx": { "gte": 1000 ,"lte": 10000 }}
                      }
              ]}}
          ],
          "must_not": { "match": {"xx": "" }}
              
          
          
        }
    },
    "collapse":{
        "field":"xxx"
    },
  "aggs": {
    "by_tags": {
      "terms": { "field": "xxx"
      },
      "aggs": {
      "sales": {
         "date_histogram": {
            "field": "xxx",
            "interval": "month", 
            "format": "yyyy-MM-dd" 
         }
      }
   }
    }
  },
  "_source":["xxx"],
  "size": 10
}

 

<11>带嵌套对象查询

参考：https://www.elastic.co/guide/cn/elasticsearch/guide/current/nested-query.html

由于嵌套对象 被索引在独立隐藏的文档中，我们无法直接查询它们。 相应地，我们必须使用 nested 查询 去获取它们：

对于nested对象的查询，需要套上一层nested

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
GET /xxxxx/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "nested": {
            "path": "t4", 
            "query": {
              "bool": {
                "must": [ 
                  {
                    "match": {
                      "t4.t1": "HelloWorld"
                    }
                  }
                ]
              }
            }
          }
        }
      ]
}}
}

或者

1
2
3
4
5
6
7
8
9
10
11
12
13
14
GET /xxxxx/_search
{
    "query": {
    "nested": {
      "path": "t4",
      "query": {
        "multi_match" : {
            "query" : "HelloWorld",
            "fields": ["t4.t1", "sex"]
        }
    }
    }}
}

 

Es优化：

Elasticsearch 技术分析（七）： Elasticsearch 的性能优化

查看索引是否关闭#

1
2
http://localhost:9200/_cat/indices/index_name?h=status

 

重建索引

因为数值类型的es字段，在query的字符串不能转换成数值的时候，需要把字段的类型从long改成keyword，先修改索引模板的字段的类型，然后执行reindex命令

1
2
3
4
5
6
7
8
9
10
POST _reindex
{
  "source": {
    "index": "twitter1"
  },
  "dest": {
    "index": "twitter1_new"
  }
}

　　

 

tonglin0325

标签

标签云

AWS Airbnb Android CDH Doris ELK Flink Git Grafana HAProxy HBase Hadoop Hive Hudi InfluxDB JDBC JMeter JVM JavaSE JavaWeb LaTex Linkedin Linux ML ML Infra MPP Maven MySQL Nexus OpenTSDB Paper Play Prometheus Python React Redis RocksDB Scala Solr Spark SpringBoot SpringMVC Thrift YARN antlr arthas avro cassandra clickhouse confluent docker filebeat flume golang google gradle hexo impala jenkins k8s kafka kerberos kudu ldap mac mongo mybatis nginx nlp open-falcon parquet presto ranger scyllaDB twitter zookeeper 前端 图存储及计算 多线程 广告系统 开发工具 数据结构 杂谈 正则表达式 算法 系统设计 计算机网络 设计模式

文章归档

最近文章