ELK

ELK 엘라스틱서치 정의 및 설치하기

1. Homebrew

Homebrew는 자유-오픈 소스 소프트웨어 패키지 관리 시스템의 하나로서 애플의 macOS 운영 체제의 소프트웨어 설치를 단순하게 만들어준다. 원래 맥스 호웰(Max Howell)이 개발한 이 패키지 관리자는 루비 온 웨일즈 공동체에서 인기를 얻었으며 확장성으로 호평을 받았습니다.홈브루는 쉬운 사용성, 명령 줄로의 통합이 가능하므로 이용이 권장됩니다.

1
$/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Homebrew를 이용하기 위해 설치를 진행해줍니다.

2. ELK란?

2.1. ElasticSearch

  • ElasticSearch는 Lucene 기반으로 개발한 분산 검색엔진으로, Logstash를 통해 수신된 데이터를 저장소에 저장하는 역할을 담당합니다.
  • 데이터를 중심부에 저장하여 예상되는 항목을 검색하고 예상치 못한 항목을 찾아낼 수 있습니다.
  • 정형, 비정형, 위치정보, 메트릭 등 원하는 방법으로 다양한 유형의 검색을 수행하고 결합할 수 있습니다.

2.2. Logstash

  • 오픈소스 서버측 데이터 처리 파이프라인으로, 다양한 소스에서 동시에 데이터를 수집하고 변환하여 stash 보관소로 보내는 역할을 합니다.
  • 수집할 로그를 선정해서, 지정된 대상 서버(ElasticSearch)에 인덱싱하여 전송하는 역할을 담당하는 소프트웨어입니다.

2.3. Kibana

  • 데이터를 시각적으로 탐색하고 실시간으로 분석 할 수 있습니다.
  • 시각화를 담당하는 HTML와 Javascript 엔진입니다.

3. Elastic 설치하기

설치전 이슈

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$brew tap elastic/tap
Error:
homebrew-core is a shallow clone.
To `brew update`, first run:
git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
This restriction has been made on GitHub's request because updating shallow
clones is an extremely expensive operation due to the tree layout and traffic of
Homebrew/homebrew-core and Homebrew/homebrew-cask. We don't do this for you
automatically to avoid repeatedly performing an expensive unshallow operation in
CI systems (which should instead be fixed to not use shallow clones). Sorry for
the inconvenience!
==> Tapping elastic/tap
Cloning into '/usr/local/Homebrew/Library/Taps/elastic/homebrew-tap'...
remote: Enumerating objects: 32, done.
remote: Counting objects: 100% (32/32), done.
remote: Compressing objects: 100% (29/29), done.
remote: Total 791 (delta 15), reused 8 (delta 3), pack-reused 759
Receiving objects: 100% (791/791), 181.90 KiB | 432.00 KiB/s, done.
Resolving deltas: 100% (601/601), done.
Tapped 17 formulae (44 files, 289.5KB).

ELK를 설치하는데 있어서 해당오류가 발생하여 다음과 같이 해결을 진행하였습니다. 업데이트가 제대로 진행이 되지 않아 homebrew core를 업데이트를 진행하였습니다.

1
2
3
4
5
6
7
8
9
10
$brew update
Error:
homebrew-core is a shallow clone.
To `brew update`, first run:
git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
This restriction has been made on GitHub's request because updating shallow
clones is an extremely expensive operation due to the tree layout and traffic of
Homebrew/homebrew-core and Homebrew/homebrew-cask. We don't do this for you
automatically to avoid repeatedly performing an expensive unshallow operation in
CI systems (which should instead be fixed to not use shallow clones). Sorry for

Homebrew를 업데이트를 진행하려고하였지만, 역시나 같은오류가 발생하여 Stackoverflow의 힘을 빌리게 되었습니다.
https://stackoverflow.com/questions/65160625/cant-update-or-upgrade-homebrew

homebrew core update

1
2
3
cd /usr/local/Homebrew/Library/Taps/homebrew/
rm -rf homebrew-core
git clone https://github.com/Homebrew/homebrew-core.git

다음과 같은 명령어로 현재 Homebrew 라이브러리안에 있는 탭볼륨의 코어를 삭제후 다시 설치를 진행하였습니다.

Elastic 설치하기

이제 본격적으로 ELK설치를 진행하겠습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
cd /usr/local/Homebrew/Library/Taps/homebrew/
rm -rf homebrew-core
git clone https://github.com/Homebrew/homebrew-core.git
Cloning into 'homebrew-core'...
remote: Enumerating objects: 34, done.
remote: Counting objects: 100% (34/34), done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 910921 (delta 19), reused 19 (delta 10), pack-reused 910887
Receiving objects: 100% (910921/910921), 364.35 MiB | 8.72 MiB/s, done.
Resolving deltas: 100% (620824/620824), done.
Updating files: 100% (5773/5773), done.
kgh  /usr/local/Homebrew/Library/Taps/homebrew   stable
$brew tap elastic/tap
Updating Homebrew...
To restore the stashed changes to /usr/local/Homebrew/Library/Taps/adoptopenjdk/homebrew-openjdk run:
'cd /usr/local/Homebrew/Library/Taps/adoptopenjdk/homebrew-openjdk && git stash pop'
==> Auto-updated Homebrew!
Updated 3 taps (homebrew/cask, homebrew/services and adoptopenjdk/openjdk).
==> New Casks
accordance forticlient-vpn mubu spotter
aldente fspy munkiadmin start
astah-uml gather opgg stringz
atemosc gdat oss-browser textbuddy
audiogridder-plugin google-drive parsify the-unofficial-homestuck-collection
audiogridder-server grid pathephone twobird
bleunlock helo pibar ultdata
brewlet imdone pokemon-trading-card-game-online unclack
cakebrewjs imobie-m1-app-checker pragli vine-server
cinco intellidock prezi-video vofa-plus
colorwell kieler ptpwebcam webull
curseforge kiwi-for-gmail ray wolfram-engine
deadbeef lightform raycast xcodes
decloner lightkey sengi xournal-plus-plus
deskreen little-navmap signet yesplaymusic
digital macfuse sitesucker-pro zy-player
duckstation mathinspector slippi-dolphin
f-bar melonds space-saver
fawkes miniforge spaceman
==> Updated Casks
Updated 1352 casks.
==> Deleted Casks
archi clashxr facebook-ios-sdk jeromelebel-mongohub racket-cs tagalicious
arrayfire disablemonitor google-hangouts oni ringtones teamspeak-client
barxtemp evom irip project-slippi-dolphin ripit yyets

Homebrew 코어를 업데이트를 진행한 후 $brew tap elastic/tap 과 같은 명령어로 볼륨 설치를 완료하게 되었습니다.
그리고 이제 elastic을 설치해보겠습니다

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
brew install elasticsearch-full
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
Updated 2 formulae.

Warning: elasticsearch has been deprecated because it is switching to an incompatible license!
==> Downloading https://homebrew.bintray.com/bottles/elasticsearch-7.10.2.cat
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/6bb47c36590116e78d
#=#=# ##O#- # ##O=# # 0. 0. 0. 0. 0. 0. 1.# 1.# 1.# 2.## 3.### 4.### 5.#### 6.##### 7.###### 8.####### 11.######## 12.######### 13.########## 15.########### 16.############ 17.############## 19.############### 21.################ 22.################# 24.################## 25.################### 27.#################### 29.##################### 30.####################### 32.######################## 34.######################### 35.########################## 36.########################### 38.############################ 40.############################# 41.############################## 42.################################ 44.################################# 46.################################## 47.################################## 48.#################################### 51.##################################### 52.##################################### 52.###################################### 53.######################################## 56.######################################## 56.######################################### 57.########################################## 59.############################################ 61.############################################# 63.############################################## 64.############################################### 65.################################################ 66.################################################# 68.################################################## 69.################################################## 70.################################################### 71.#################################################### 73.#################################################### 73.##################################################### 74.####################################################### 76.######################################################## 78.######################################################## 78.########################################################## 81.########################################################### 82.############################################################ 84.############################################################# 86.############################################################### 87.################################################################ 88.################################################################# 90.################################################################## 92.#################################################################### 94.##################################################################### 96.##################################################################### 97.####################################################################### 98.######################################################################## 100.0%
==> Pouring elasticsearch-7.10.2.catalina.bottle.tar.gz
==> /usr/local/Cellar/elasticsearch/7.10.2/bin/elasticsearch-keystore create
==> Caveats
Data: /usr/local/var/lib/elasticsearch/
Logs: /usr/local/var/log/elasticsearch/elasticsearch_kgh.log
Plugins: /usr/local/var/elasticsearch/plugins/
Config: /usr/local/etc/elasticsearch/

To have launchd start elasticsearch now and restart at login:
brew services start elasticsearch
Or, if you don't want/need a background service you can just run:
elasticsearch
==> Summary
🍺 /usr/local/Cellar/elasticsearch/7.10.2: 156 files, 113.5MB

4. Kibana 설치하기

다음과 같은 명령어로 Kibana설치를 진행합니다

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
brew install elastic/tap/kibana(full을 사용하지않으면 최신화적인 elasticsearch와의 버전문제가 생길 가능성이 있습니다.) 사용 X

brew install elastic/tap/kibana-full(사용O)
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
Updated 2 formulae.

==> Installing kibana-full from elastic/tap
==> Downloading https://artifacts.elastic.co/downloads/kibana/kibana-7.11.1-darwin-x86_64.tar.gz?tap=elastic/homebrew-tap
######################################################################## 100.0%
==> Caveats
Config: /usr/local/etc/kibana/
If you wish to preserve your plugins upon upgrade, make a copy of
/usr/local/opt/kibana-full/plugins before upgrading, and copy it into the
new keg location after upgrading.

To have launchd start elastic/tap/kibana-full now and restart at login:
brew services start elastic/tap/kibana-full
Or, if you don't want/need a background service you can just run:
kibana
==> Summary
🍺 /usr/local/Cellar/kibana-full/7.11.1: 47,465 files, 655.5MB, built in 2 minutes 10 seconds
==> `brew cleanup` has not been run in 30 days, running now...
Removing: /Users/kgh/Library/Caches/Homebrew/bdw-gc--8.0.4_1.catalina.bottle.1.tar.gz... (513KB)
Removing: /Users/kgh/Library/Caches/Homebrew/certbot--1.10.1.catalina.bottle.tar.gz... (7.2MB)
Removing: /Users/kgh/Library/Caches/Homebrew/dialog--1.3-20201126.catalina.bottle.tar.gz... (330.7KB)
Removing: /Users/kgh/Library/Caches/Homebrew/emacs--27.1.catalina.bottle.tar.gz... (44.3MB)
Removing: /usr/local/Cellar/gdbm/1.18.1... (20 files, 591.2KB)
Removing: /Users/kgh/Library/Caches/Homebrew/gnu-getopt--2.36.1.catalina.bottle.tar.gz... (56.2KB)
Removing: /Users/kgh/Library/Caches/Homebrew/go--1.15.6.catalina.bottle.tar.gz... (153.9MB)
Removing: /Users/kgh/Library/Caches/Homebrew/groonga--10.1.0.catalina.bottle.tar.gz... (8MB)
Removing: /Users/kgh/Library/Caches/Homebrew/guile--3.0.4.catalina.bottle.2.tar.gz... (11.5MB)
Removing: /Users/kgh/Library/Caches/Homebrew/mariadb--10.5.8.catalina.bottle.2.tar.gz... (50.6MB)
Removing: /Users/kgh/Library/Caches/Homebrew/nettle--3.6.catalina.bottle.tar.gz... (866.6KB)
Removing: /Users/kgh/Library/Caches/Homebrew/nghttp2--1.42.0_1.catalina.bottle.tar.gz... (971.0KB)
Removing: /Users/kgh/Library/Caches/Homebrew/node--15.5.0.catalina.bottle.tar.gz... (15.1MB)
Removing: /Users/kgh/Library/Caches/Homebrew/openjdk--15.0.1.catalina.bottle.tar.gz... (190.9MB)
Removing: /usr/local/Cellar/pkg-config/0.29.2... (11 files, 627.2KB)
Removing: /Users/kgh/Library/Caches/Homebrew/python@3.9--setuptools--51.0.0.zip... (2.0MB)
Removing: /Users/kgh/Library/Caches/Homebrew/python@3.9--pip--20.3.1.tar.gz... (1.5MB)
Removing: /Users/kgh/Library/Caches/Homebrew/python@3.9--wheel--0.36.1.tar.gz... (62.7KB)
Removing: /Users/kgh/Library/Caches/Homebrew/redis--6.0.9.catalina.bottle.tar.gz... (1.8MB)
Removing: /Users/kgh/Library/Caches/Homebrew/sphinx-doc--3.4.1.catalina.bottle.tar.gz... (15.1MB)
Removing: /Users/kgh/Library/Caches/Homebrew/unbound--1.13.0.catalina.bottle.tar.gz... (2.7MB)
Removing: /Users/kgh/Library/Logs/Homebrew/pkg-config... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/tree... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/libtool... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/jansson... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/gmp... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/msgpack... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/augeas... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/libidn2... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/nghttp2... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/go... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/nettle... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/mariadb... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/gdbm... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/gnu-getopt... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/libunistring... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/jemalloc... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/bdw-gc... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/maven... (101B)
Removing: /Users/kgh/Library/Logs/Homebrew/icu4c... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/certbot... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/readline... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/sqlite... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/c-ares... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/yarn... (100B)
Removing: /Users/kgh/Library/Logs/Homebrew/nvm... (99B)
Removing: /Users/kgh/Library/Logs/Homebrew/emacs... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/dialog... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/openjdk... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/git-flow... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/gettext... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/git-flow-avh... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/sphinx-doc... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/libtasn1... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/libevent... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/openssl@1.1... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/cask... (100B)
Removing: /Users/kgh/Library/Logs/Homebrew/node... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/unbound... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/zsh... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/groonga... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/guile... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/p11-kit... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/libev... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/protobuf... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/libffi... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/python@3.9... (2 files, 754.7KB)
Removing: /Users/kgh/Library/Logs/Homebrew/ethereum... (64B)
Removing: /Users/kgh/Library/Logs/Homebrew/gnutls... (64B)
Pruned 2 symbolic links and 3 directories from /usr/local

5. Elasticsearch 실행하기

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
elasticsearch
warning: no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
warning: no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
[2021-02-19T12:05:59,584][INFO ][o.e.n.Node ] [kgh-2.local] version[7.10.2-SNAPSHOT], pid[94450], build[oss/tar/unknown/2021-01-16T01:34:41.142971Z], OS[Mac OS X/10.15.5/x86_64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-02-19T12:05:59,590][INFO ][o.e.n.Node ] [kgh-2.local] JVM home [/usr/local/Cellar/openjdk/15.0.1/libexec/openjdk.jdk/Contents/Home]
[2021-02-19T12:05:59,598][INFO ][o.e.n.Node ] [kgh-2.local] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/folders/fy/5zsq88p55dx45pk4gmpltmmr0000gp/T/elasticsearch-9946170376789376085, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/usr/local/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/local/Cellar/elasticsearch/7.10.2/libexec, -Des.path.conf=/usr/local/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=false]
[2021-02-19T12:05:59,599][WARN ][o.e.n.Node ] [kgh-2.local] version [7.10.2-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
[2021-02-19T12:06:00,853][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [aggs-matrix-stats]
[2021-02-19T12:06:00,855][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [analysis-common]
[2021-02-19T12:06:00,856][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [geo]
[2021-02-19T12:06:00,857][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [ingest-common]
[2021-02-19T12:06:00,858][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [ingest-geoip]
[2021-02-19T12:06:00,859][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [ingest-user-agent]
[2021-02-19T12:06:00,859][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [kibana]
[2021-02-19T12:06:00,860][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [lang-expression]
[2021-02-19T12:06:00,861][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [lang-mustache]
[2021-02-19T12:06:00,861][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [lang-painless]
[2021-02-19T12:06:00,862][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [mapper-extras]
[2021-02-19T12:06:00,863][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [parent-join]
[2021-02-19T12:06:00,864][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [percolator]
[2021-02-19T12:06:00,864][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [rank-eval]
[2021-02-19T12:06:00,865][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [reindex]
[2021-02-19T12:06:00,865][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [repository-url]
[2021-02-19T12:06:00,866][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [test-delayed-aggs]
[2021-02-19T12:06:00,867][INFO ][o.e.p.PluginsService ] [kgh-2.local] loaded module [transport-netty4]
[2021-02-19T12:06:00,868][INFO ][o.e.p.PluginsService ] [kgh-2.local] no plugins loaded
[2021-02-19T12:06:00,920][INFO ][o.e.e.NodeEnvironment ] [kgh-2.local] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s1)]], net usable_space [39.2gb], net total_space [233.5gb], types [apfs]
[2021-02-19T12:06:00,921][INFO ][o.e.e.NodeEnvironment ] [kgh-2.local] heap size [1gb], compressed ordinary object pointers [true]
[2021-02-19T12:06:01,012][INFO ][o.e.n.Node ] [kgh-2.local] node name [kgh-2.local], node ID [_FYb0BAnQFyCJd4z4-M3Jw], cluster name [elasticsearch_brew], roles [master, remote_cluster_client, data, ingest]
[2021-02-19T12:06:05,212][INFO ][o.e.t.NettyAllocator ] [kgh-2.local] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2021-02-19T12:06:05,377][INFO ][o.e.d.DiscoveryModule ] [kgh-2.local] using discovery type [zen] and seed hosts providers [settings]
[2021-02-19T12:06:05,870][WARN ][o.e.g.DanglingIndicesState] [kgh-2.local] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-02-19T12:06:06,195][INFO ][o.e.n.Node ] [kgh-2.local] initialized
[2021-02-19T12:06:06,196][INFO ][o.e.n.Node ] [kgh-2.local] starting ...
[2021-02-19T12:06:06,446][INFO ][o.e.t.TransportService ] [kgh-2.local] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-02-19T12:06:06,883][WARN ][o.e.b.BootstrapChecks ] [kgh-2.local] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-02-19T12:06:06,944][INFO ][o.e.c.c.ClusterBootstrapService] [kgh-2.local] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-02-19T12:06:09,949][INFO ][o.e.c.c.Coordinator ] [kgh-2.local] setting initial configuration to VotingConfiguration{_FYb0BAnQFyCJd4z4-M3Jw}
[2021-02-19T12:06:10,258][INFO ][o.e.c.s.MasterService ] [kgh-2.local] elected-as-master ([1] nodes joined)[{kgh-2.local}{_FYb0BAnQFyCJd4z4-M3Jw}{XfIeFRDrTIqm91ZOBkXD1Q}{127.0.0.1}{127.0.0.1:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{kgh-2.local}{_FYb0BAnQFyCJd4z4-M3Jw}{XfIeFRDrTIqm91ZOBkXD1Q}{127.0.0.1}{127.0.0.1:9300}{dimr}]}
[2021-02-19T12:06:10,362][INFO ][o.e.c.c.CoordinationState] [kgh-2.local] cluster UUID set to [opZ_8jGaS-yI2qclxNlqhg]
[2021-02-19T12:06:10,433][INFO ][o.e.c.s.ClusterApplierService] [kgh-2.local] master node changed {previous [], current [{kgh-2.local}{_FYb0BAnQFyCJd4z4-M3Jw}{XfIeFRDrTIqm91ZOBkXD1Q}{127.0.0.1}{127.0.0.1:9300}{dimr}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2021-02-19T12:06:10,487][INFO ][o.e.h.AbstractHttpServerTransport] [kgh-2.local] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2021-02-19T12:06:10,488][INFO ][o.e.n.Node ] [kgh-2.local] started
[2021-02-19T12:06:10,547][INFO ][o.e.g.GatewayService ] [kgh-2.local] recovered [0] indices into cluster_state

다음과 같이 elasticsearch 명령어로 Elasticsearch가 실행된것을 알 수 있습니다. 이제 publish_address인 127.0.0.1:9200 경로로 웹브라우저 상에서 접근을 하게 되면 다음과 같은 메시지를 얻을 수 있습니다. 아니면 아래와 같은 방식으로 curl -XGET 방식으로 직접터미널에서 확인해보는 방식이 있습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$curl -XGET localhost:9200
{
"name" : "kgh-2.local",
"cluster_name" : "elasticsearch_brew",
"cluster_uuid" : "opZ_8jGaS-yI2qclxNlqhg",
"version" : {
"number" : "7.10.2-SNAPSHOT",
"build_flavor" : "oss",
"build_type" : "tar",
"build_hash" : "unknown",
"build_date" : "2021-01-16T01:34:41.142971Z",
"build_snapshot" : true,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"name" : "kgh-2.local",
"cluster_name" : "elasticsearch_brew",
"cluster_uuid" :
"version" : {
"number" : "7.10.2-SNAPSHOT",
"build_flavor" : "oss",
"build_type" : "tar",
"build_hash" : "unknown",
"build_date" : "2021-01-16T01:34:41.142971Z",
"build_snapshot" : true,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

6. Kibana 실행하기

Terminal상에서 다음과 같은 명령어로 Kibana를 실행하여줍니다. 실행시 오류가 발생하여 보니 elasticsearch와의 버전문제가 발생하였습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$kibana
log [12:12:04.315] [info][plugins-service] Plugin "visTypeXy" is disabled.
log [12:12:04.439] [warning][config][deprecation] Environment variable "DATA_PATH" will be removed. It has been replaced with kibana.yml setting "path.data"
log [12:12:04.440] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
log [12:12:04.754] [info][plugins-system] Setting up [101] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,bfetch,share,legacyExport,embeddable,uiActionsEnhanced,expressions,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeTable,visTypeMarkdown,visTypeVega,visTypeTimelion,features,licenseManagement,dataEnhanced,tileMap,mapsOss,regionMap,lensOss,inputControlVis,graph,timelion,dashboard,dashboardEnhanced,visualize,charts,visTypeVislib,visTypeTimeseries,visTypeTimeseriesEnhanced,visTypeMetric,visTypeTagcloud,watcher,canvas,discover,discoverEnhanced,savedObjectsManagement,spaces,security,reporting,dashboardMode,savedObjectsTagging,lens,maps,lists,encryptedSavedObjects,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,remoteClusters,crossClusterReplication,rollup,indexLifecycleManagement,enterpriseSearch,ml,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,securitySolution,case,infra,monitoring,logstash,apm,uptime,esUiShared]
log [12:12:04.758] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: 75bba0a2-d69a-4376-b540-34d359878b57
log [12:12:05.076] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:12:05.077] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.
log [12:12:05.120] [warning][config][plugins][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:12:05.121] [info][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, and is supported for Darwin OS. Automatically enabling Chromium sandbox.
log [12:12:05.154] [warning][config][encryptedSavedObjects][plugins] Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:12:05.184] [warning][fleet][plugins] Fleet APIs are disabled because the Encrypted Saved Objects plugin uses an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:12:05.287] [warning][actions][actions][plugins] APIs are disabled because the Encrypted Saved Objects plugin uses an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:12:05.303] [warning][alerting][alerts][plugins][plugins] APIs are disabled because the Encrypted Saved Objects plugin uses an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:12:05.446] [info][monitoring][monitoring][plugins] config sourced from: production cluster
log [12:12:05.705] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
log [12:12:05.790] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to [illegal_argument_exception] request [/_xpack] contains unrecognized parameter: [accept_enterprise] :: {"path":"/_xpack?accept_enterprise=true","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"request [/_xpack] contains unrecognized parameter: [accept_enterprise]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"request [/_xpack] contains unrecognized parameter: [accept_enterprise]\"},\"status\":400}"} error
log [12:12:05.794] [warning][monitoring][monitoring][plugins] X-Pack Monitoring Cluster Alerts will not be available: X-Pack plugin is not installed on the Elasticsearch cluster.
log [12:12:05.900] [error][savedobjects-service] This version of Kibana (v7.11.1) is incompatible with the following Elasticsearch nodes in your cluster: v7.10.2 @ 127.0.0.1:9200 (127.0.0.1)
log [12:12:35.729] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to [illegal_argument_exception] request [/_xpack] contains unrecognized parameter: [accept_enterprise] :: {"path":"/_xpack?accept_enterprise=true","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"request [/_xpack] contains unrecognized parameter: [accept_enterprise]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"request [/_xpack] contains unrecognized parameter: [accept_enterprise]\"},\"status\":400}"} error
^C log [12:12:42.392] [info][plugins-system] Stopping all plugins.
log [12:12:42.395] [info][kibana-monitoring][monitoring][monitoring][plugins] Monitoring stats collection is stopped
^C log [12:12:42.624] [info][plugins-system] Stopping all plugins.
log [12:12:42.628] [info][savedobjects-service] Starting saved objects migrations
log [12:12:42.631] [warning][savedobjects-service] Unable to connect to Elasticsearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request.
log [12:12:42.632] [warning][savedobjects-service] Unable to connect to Elasticsearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request.
^C%
kgh  ~
$kibana --version
7.11.1

이렇게 해당오류가 발생한 이유를 보니 버전을 full 버전(즉, 최신버전으로)다운받지 않아서 이러한 호환성문제가 생기게 되었습니다.

1
2
brew install elasticsearch(최신 버전 호환 X)
brew install elasticsearch-full(최신 버전 호환 O)

따라서 다음과 같은방법으로 elasticsearch버전을 unlink해주고 새롭게 다시 설치를 진행하였습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$brew uninstall elasticsearch
$rm -rf /usr/local/etc/elasticsearch
$rm -rf /usr/local/var/lib/elasticsearch
$brew unlink elasticsearch
Unlinking /usr/local/Cellar/elasticsearch/7.10.2... 4 symlinks removed.

brew install elasticsearch-full
==> Installing elasticsearch-full from elastic/tap
==> Downloading https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.11.1-darwin-x86_64.tar.gz?tap=elastic/homebrew-tap
Already downloaded: /Users/kgh/Library/Caches/Homebrew/downloads/4c04176d64640de7c66954a6e6c536ac2ff45a342828e8af1d13fc8bf0ed47c8--elasticsearch-7.11.1-darwin-x86_64.tar.gz
Warning: Tried to install empty array to /usr/local/etc/elasticsearch/jvm.options.d
==> codesign -f -s - /usr/local/Cellar/elasticsearch-full/7.11.1/libexec/modules/x-pack-ml/platform/darwin-x86_64/controller.app --deep
==> find /usr/local/Cellar/elasticsearch-full/7.11.1/libexec/jdk.app/Contents/Home/bin -type f -exec codesign -f -s - {} ;
==> Caveats
Data: /usr/local/var/lib/elasticsearch/elasticsearch_kgh/
Logs: /usr/local/var/log/elasticsearch/elasticsearch_kgh.log
Plugins: /usr/local/var/elasticsearch/plugins/
Config: /usr/local/etc/elasticsearch/

To have launchd start elastic/tap/elasticsearch-full now and restart at login:
brew services start elastic/tap/elasticsearch-full
Or, if you don't want/need a background service you can just run:
elasticsearch
==> Summary
🍺 /usr/local/Cellar/elasticsearch-full/7.11.1: 933 files, 488.2MB, built in 19 seconds

다음과 같이 버전을 새로 설치하여 호환성을 맞추어주었습니다.

반드시 $elasticsearch가 구동되어있다면 재시작을 진행해야 정상적으로 수행이됩니다

kibana 다시 재가동하기

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
kibana
log [12:50:29.122] [info][plugins-service] Plugin "visTypeXy" is disabled.
log [12:50:29.357] [warning][config][deprecation] Environment variable "DATA_PATH" will be removed. It has been replaced with kibana.yml setting "path.data"
log [12:50:29.359] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
log [12:50:29.635] [info][plugins-system] Setting up [101] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeTimeseries,visTypeTimeseriesEnhanced,visTypeVega,visTypeTable,visTypeTimelion,features,licenseManagement,dataEnhanced,watcher,canvas,visTypeTagcloud,visTypeMetric,visTypeMarkdown,tileMap,regionMap,lensOss,mapsOss,inputControlVis,graph,timelion,dashboard,dashboardEnhanced,visualize,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,remoteClusters,crossClusterReplication,rollup,indexLifecycleManagement,enterpriseSearch,ml,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,securitySolution,case,infra,monitoring,logstash,apm,uptime]
log [12:50:29.639] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: 75bba0a2-d69a-4376-b540-34d359878b57
log [12:50:30.070] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:50:30.070] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.
log [12:50:30.180] [warning][config][plugins][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:50:30.183] [info][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, and is supported for Darwin OS. Automatically enabling Chromium sandbox.
log [12:50:30.192] [warning][config][encryptedSavedObjects][plugins] Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:50:30.265] [warning][fleet][plugins] Fleet APIs are disabled because the Encrypted Saved Objects plugin uses an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:50:30.393] [warning][actions][actions][plugins] APIs are disabled because the Encrypted Saved Objects plugin uses an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:50:30.414] [warning][alerting][alerts][plugins][plugins] APIs are disabled because the Encrypted Saved Objects plugin uses an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [12:50:30.540] [info][monitoring][monitoring][plugins] config sourced from: production cluster
log [12:50:30.816] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
log [12:50:31.036] [info][savedobjects-service] Starting saved objects migrations
log [12:50:31.095] [info][savedobjects-service] Creating index .kibana_task_manager_1.
log [12:50:31.100] [info][savedobjects-service] Creating index .kibana_1.
log [12:50:31.929] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1.
log [12:50:32.013] [info][savedobjects-service] Pointing alias .kibana to .kibana_1.
log [12:50:32.134] [info][savedobjects-service] Finished in 1039ms.
log [12:50:32.188] [info][savedobjects-service] Finished in 1101ms.
log [12:50:32.225] [info][plugins-system] Starting [101] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeTimeseries,visTypeTimeseriesEnhanced,visTypeVega,visTypeTable,visTypeTimelion,features,licenseManagement,dataEnhanced,watcher,canvas,visTypeTagcloud,visTypeMetric,visTypeMarkdown,tileMap,regionMap,lensOss,mapsOss,inputControlVis,graph,timelion,dashboard,dashboardEnhanced,visualize,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,remoteClusters,crossClusterReplication,rollup,indexLifecycleManagement,enterpriseSearch,ml,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,securitySolution,case,infra,monitoring,logstash,apm,uptime]
log [12:50:35.778] [info][listening] Server running at http://localhost:5601
log [12:50:36.837] [info][server][Kibana][http] http server running at http://localhost:5601
log [12:50:36.851] [info][plugins][watcher] Your basic license does not support watcher. Please upgrade your license.
log [12:50:36.865] [info][crossClusterReplication][plugins] Your basic license does not support crossClusterReplication. Please upgrade your license.
log [12:50:36.876] [info][kibana-monitoring][monitoring][monitoring][plugins] Starting monitoring stats collection
log [12:50:39.515] [info][plugins][reporting] Downloading https://storage.googleapis.com/headless_shell/chromium-ef768c9-darwin_x64.zip to /usr/local/Cellar/kibana-full/7.11.1/libexec/.chromium/chromium-ef768c9-darwin_x64.zip
log [12:50:39.517] [info][plugins][reporting] Downloading https://storage.googleapis.com/headless_shell/chromium-ef768c9-linux_x64.zip to /usr/local/Cellar/kibana-full/7.11.1/libexec/.chromium/chromium-ef768c9-linux_x64.zip
log [12:50:39.518] [info][plugins][reporting] Downloading https://storage.googleapis.com/headless_shell/chromium-ef768c9-linux_arm64.zip to /usr/local/Cellar/kibana-full/7.11.1/libexec/.chromium/chromium-ef768c9-linux_arm64.zip
log [12:50:39.519] [info][plugins][reporting] Downloading https://storage.googleapis.com/headless_shell/chromium-ef768c9-windows_x64.zip to /usr/local/Cellar/kibana-full/7.11.1/libexec/.chromium/chromium-ef768c9-windows_x64.zip
log [12:51:03.269] [info][plugins][reporting] Downloaded https://storage.googleapis.com/headless_shell/chromium-ef768c9-linux_x64.zip
log [12:51:05.427] [info][plugins][reporting] Downloaded https://storage.googleapis.com/headless_shell/chromium-ef768c9-darwin_x64.zip
log [12:51:06.716] [info][plugins][reporting] Downloaded https://storage.googleapis.com/headless_shell/chromium-ef768c9-windows_x64.zip
log [12:51:07.962] [info][plugins][reporting] Downloaded https://storage.googleapis.com/headless_shell/chromium-ef768c9-linux_arm64.zip
log [12:51:07.969] [info][plugins][reporting] Extracting [/usr/local/Cellar/kibana-full/7.11.1/libexec/.chromium/chromium-ef768c9-darwin_x64.zip] to [/usr/local/Cellar/kibana-full/7.11.1/libexec/x-pack/plugins/reporting/chromium/headless_shell-darwin_x64/headless_shell]
log [12:51:07.995] [warning][environment] Detected an unhandled Promise rejection.
Error: Cannot delete files/directories outside the current working directory. Can be overridden with the `force` option.

버전별 설치하기

1
2
curl -O https://artifacts.elastic.co/downloads/kibana/kibana-7.10.2-darwin-x86_64.tar.gz
tar -xzf kibana-7.10.2-darwin-x86_64.tar.gz

https://smartstuartkim.wordpress.com/2019/09/23/install-elastic-stack-on-mac-session-1-openjdk-elastic-search-kibana-logstash-beat/

7. kibana 가동

이제 다음과 같이 구동이되면 정상적으로 실행이 된것을 알 수 있습니다.

Kibana