CloudWatch でアプリケーションのログを取得

環境

software version
ubuntu 18.04 LTS

CloudWatch Logs Agent の導入

CloudWatch でアプリケーションのログを取得するには、対象となるサーバに CloudWatch Logs Agent を導入する。

ubuntu@ip-10-0-0-183:~$ curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 56093  100 56093    0     0  70824      0 --:--:-- --:--:-- --:--:-- 70735
ubuntu@ip-10-0-0-183:~$
ubuntu@ip-10-0-0-183:~$
ubuntu@ip-10-0-0-183:~$ python -V
Python 2.7.15rc1
ubuntu@ip-10-0-0-183:~$
ubuntu@ip-10-0-0-183:~$ python3 -V
Python 3.6.7

python3 でインストールしようとしたら怒られた。

ubuntu@ip-10-0-0-183:~$ sudo python3 ./awslogs-agent-setup.py --region us-east-1
ERROR: This script only supports python version 2.6 - 3.5

python でインストール。

ubuntu@ip-10-0-0-183:~$ sudo python ./awslogs-agent-setup.py --region us-east-1
Launching interactive setup of CloudWatch Logs agent ...

Step 1 of 5: Installing pip ...libyaml-dev does not exist in system DONE

Step 2 of 5: Downloading the latest CloudWatch Logs agent bits ... DONE

Step 3 of 5: Configuring AWS CLI ...
AWS Access Key ID [****************LIIA]:
AWS Secret Access Key [****************pQqi]:
Default region name [us-east-1]: ap-northeast-1
Default output format [None]: json

Step 4 of 5: Configuring the CloudWatch Logs Agent ...
Path of log file to upload [/var/log/syslog]:
Destination Log Group name [/var/log/syslog]:

Choose Log Stream name:
  1. Use EC2 instance id.
  2. Use hostname.
  3. Custom.
Enter choice [1]:

Choose Log Event timestamp format:
  1. %b %d %H:%M:%S    (Dec 31 23:59:59)
  2. %d/%b/%Y:%H:%M:%S (10/Oct/2000:13:55:36)
  3. %Y-%m-%d %H:%M:%S (2008-09-08 11:52:54)
  4. Custom
Enter choice [1]: 3

Choose initial position of upload:
  1. From start of file.
  2. From end of file.
Enter choice [1]:
More log files to configure? [Y]: n

Step 5 of 5: Setting up agent as a daemon ...DONE


------------------------------------------------------
- Configuration file successfully saved at: /var/awslogs/etc/awslogs.conf
- You can begin accessing new log events after a few moments at https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logs:
- You can use 'sudo service awslogs start|stop|status|restart' to control the daemon.
- To see diagnostic information for the CloudWatch Logs Agent, see /var/log/awslogs.log
- You can rerun interactive setup using 'sudo python ./awslogs-agent-setup.py --region us-east-1 --only-generate-config'
------------------------------------------------------
ubuntu@ip-10-0-0-183:~$

CloudWatch Logs Agent の設定確認

/var/awslogs/etc/awslogs.confが保管先になる。

$ sudo cat /var/awslogs/etc/awslogs.conf

(途中省略)

[/var/log/syslog]
datetime_format = %Y-%m-%d %H:%M:%S
file = /var/log/syslog
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/syslog

errorlogを意図的に出力させたかったのでアクセスを許可しないファイルを用意。

ubuntu@ip-10-0-0-183:/var/www/html$ ll
total 20
drwxr-xr-x 2 root root  4096 May 21 14:30 ./
drwxr-xr-x 3 root root  4096 May 21 14:09 ../
-r-------- 1 root root     0 May 21 14:30 denied.html
-rw-r--r-- 1 root root 10918 May 21 14:09 index.html
ubuntu@ip-10-0-0-183:/var/www/html$

ブラウザからアクセスしてエラーが出力されることを確認。

ubuntu@ip-10-0-0-183:/var/www/html$ tail /var/log/apache2/error.log
[Tue May 21 14:09:06.072766 2019] [mpm_event:notice] [pid 25946:tid 140322336525248] AH00489: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Tue May 21 14:09:06.072862 2019] [core:notice] [pid 25946:tid 140322336525248] AH00094: Command line: '/usr/sbin/apache2'
[Tue May 21 14:31:34.997717 2019] [core:error] [pid 25948:tid 140322109630208] (13)Permission denied: [client 180.45.162.2:63040] AH00132: file permissions deny server access: /var/www/html/denied.html

CloudWatch Logs Agent を再起動。

ubuntu@ip-10-0-0-183:/var/www/html$ sudo systemctl restart awslogs

再度ブラウザからアクセスしてapacheのエラーを出力。 しかし、CloudWatch Logs でログが確認出来ず。

CloudWatch Logs Agent のログを確認。

ubuntu@ip-10-0-0-183:~$ sudo tail /var/log/awslogs.log

(省略)

ClientError: An error occurred (AccessDeniedException) when calling the PutLogEvents operation: User: arn:aws:iam::123456789012:user/iamuser_name is not authorized to perform: logs:PutLogEvents on resource: arn:aws:logs:ap-northeast-1:123456789012:log-group:/var/log/apache2/error.log:log-stream:i-xxxxxxxxxxxxxxxxxxx
2019-05-21 15:29:30,566 - cwlogs.push.reader - WARNING - 28881 - Thread-4 - Fall back to previous event time: {'timestamp': 1558420113000, 'start_position': 56897L, 'end_position': 56979L}, previousEventTime: 1558420113000, reason: timestamp could not be parsed from message.

検証していた環境では awscli の credentials を登録していたため、設定したEC2ロールではなく awscli の credentials 情報を使用していた様子。 awscli の credentials 情報を削除するとエラーが解消された。

CloudWatch のログメトリクスにも出力されるようになった。

f:id:dafukui:20190521155254p:plain