刘凡 9ff4d1d109 add S3,archive,truncate | 2 年 前 | |
---|---|---|
.. | ||
LICENSE | 2 年 前 | |
README.md | 2 年 前 | |
to_parquet.py | 2 年 前 |
Simple python script to convert Zeek ascii logs to parquet format and upload to Amazon S3
Install dependencies
pip install zat awscli fastparquet s3fs
Add your credentials
aws configure
Edit /opt/zeek/share/zeekctl/scripts/archive-log
script (let me know if there is a better way). I've placed this script just before archive-log deletes
the logfile.
158
159 # convert zeek logs to parquet and uploda to s3
160 /usr/bin/python3 /root/to_parquet.py $file_name $base_name s3://zeek.threatbear.co/
161
162 rm -f $file_name
NOTE : Make sure you put the trailing slash on the s3://zeek.threatbear.co/
URL otherwise the script will fail.
IMPORTANT: Zeek package updates will overwrite /opt/zeek/share/zeekctl/scripts/archive-log
so until we find another way of running zeek postprocessors YOU WILL NEED TO MAKE THE EDIT AFTER EACH UPDATE!!
Restart Zeek
zeekctl restart
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::zeek.threatbear.co/*"
}
]
}