AIS Projects / logstatalarm

Download as
Browse Code »

Commit adfda8e12384c9f984db9762dd852b7a810399e7

Authored by Arsisakarn Srilatanart
0 parents
Exists in master and in 1 other branch original-fix-stat

initial commit from ais code

Showing 16 changed files with 1051 additions and 0 deletions   Show diff stats
.gitignore 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/.gitignore
... ... @@ -0,0 +1,4 @@
  1 +log_test/
  2 +whole_flow/
  3 +node_modules/
  4 +.DS_Store
... ...
READ.ME 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/READ.ME
... ... @@ -0,0 +1,182 @@
  1 +# LOG STAT ALARM
  2 +
  3 + ### Log
  4 +
  5 + * Append JSON object, array, or string to a log file
  6 + * All type will be converted to string automatically
  7 + * ISO string and unix time will be with each log by default
  8 +
  9 + ### Stat
  10 +
  11 + * Keep track of value within the applications
  12 + * Call 'increment' method of specific value whenever incrementation is needed
  13 + * Stat will automatically append all value into a file when it reaches its configured interval
  14 + * Value can be pre-configured for alarm (threshold, inverted threshold: read more on Alarm section)
  15 + * ISO string and unix time will be with each stat by default
  16 +
  17 + ### Alarm
  18 +
  19 + * If any stat value exceed its pre-configured threshold, it will be logged into alarm file
  20 + * If any stat value does not reach its pre-configured inverted threshold, it will also be logged into alarm file
  21 + * Before logging into alarm file, the module will call external functions that are being provided on init method (more on example)
  22 + * Stat module will consistently check the above rules in at the end of each interval
  23 +
  24 + ### File Management
  25 +
  26 + * Every module has option to configure file rotation frequency
  27 + * File rotation can be configured in two ways:
  28 + * Rotate by time (in millisecond)
  29 + * Rotate by file max size (in byte)
  30 + * Each module can be configured with both options ( read more on example section )
  31 +
  32 + ### Example
  33 +
  34 + ##### Initialization
  35 +
  36 + Initialization should be called only once in each application
  37 + Use the response object for log stat alarm
  38 +
  39 + Options
  40 + * dirname: the parent folder for log, stat, and alarm ('./logstatalarm'). Log, stat and alarm files will be auto generated as child folders ( Type: string )
  41 + * ./logstatalarm/log/
  42 + * ./logstatalarm/stat/
  43 + * ./logstatalarm/alarm/
  44 + * log.rotation: seperate files into time frame ( Type: integer (ms) )
  45 + * 1000 = 1 second
  46 + * 60 * 1000 = 60 seconds
  47 + * 15 * 60 * 1000 = 15 minutes
  48 + * log.maxsize: if the current time frame for the file exceed maxsize, it will auto generate file with same time frame with suffix to count ( Type: integer (byte) )
  49 + * 1 = 1 byte
  50 + * 1000 = 1 Kb
  51 + * 1000000 = 1 Mb
  52 + * stat.rotation: seperate files into time frame ( Type: integer (ms) )
  53 + * 1000 = 1 second
  54 + * 60 * 1000 = 60 seconds
  55 + * 15 * 60 * 1000 = 15 minutes
  56 + * stat.maxsize: if the current time frame for the file exceed maxsize, it will auto generate file with same time frame with suffix to count ( Type: integer (byte) )
  57 + * 1 = 1 byte
  58 + * 1000 = 1 Kb
  59 + * 1000000 = 1 Mb
  60 + * stat.interval: interval defines how frequent stat should be append to file. All value counter will be reset to 0 and if any value is not within the configured threshold range, it will be log to alarm file. ( Type: interger (ms) )
  61 + * 1000 = 1 second
  62 + * 2 * 1000 = 2 seconds
  63 + * 2 * 60 * 1000 = 2 minutes
  64 + * stat.data: Data can be pre-configured with threshold and inverted threshold ( Type: array of object )
  65 + * Object data type: { key: string, threshold: integer (min 0), threshold_inv: integer (min 0) }
  66 + * Threshold and threshold_inv is not required
  67 + * At the end of each interval, stat value will be checked if it exceed threshold or haven't reach threshold_inv. If so, it will log to alarm file automatically
  68 + * alarm.rotation: seperate files into time frame ( Type: integer (ms) )
  69 + * 1000 = 1 second
  70 + * 60 * 1000 = 60 seconds
  71 + * 15 * 60 * 1000 = 15 minutes
  72 + * alarm.maxsize: if the current time frame for the file exceed maxsize, it will auto generate file with same time frame with suffix to count ( Type: integer (byte) )
  73 + * 1 = 1 byte
  74 + * 1000 = 1 Kb
  75 + * 1000000 = 1 Mb
  76 + * alarm.external: ( Type: Array )
  77 + * alarm.external[].fn: ( Type: Function)
  78 + * This function will be called right before the module log into alarm log file (Use cases: http request, snmp request, syslog file logging, etc.)
  79 + * All functions will be called asynchronously
  80 + * All callback function within these functions will be ignored
  81 + * All return data within these functions will also be ignored
  82 + * Make sure the first argument of this function is for data (This data is the same data that logged in to alarm file))
  83 + * Error handling have to be carefully implemented because all return data will be ignored
  84 + * alarm.external[].args: ( Type: Array )
  85 + * alarm.external[].args[]: arguments for function above ( Type: Any )
  86 + * Args will be use for executing the alarm.external[].fn function (more on examples)
  87 +
  88 +
  89 + Note*: In this example, we will be using request module (npm install request) on alarm external function for demonstration purpose only.
  90 + ```
  91 + var logstatalarm = require('logstatalarm');
  92 + var request = require('request');
  93 + var http_request = function(data, host, port, path, method) { // make sure your first argument is Data
  94 + var opts = {
  95 + host: host,
  96 + port: port,
  97 + path: path,
  98 + method: method,
  99 + body: data // Data will always be in type: String
  100 + request(opts, function(err, res) { // Any callback or return data will be ignore
  101 + if(err) {
  102 + console.log(err);
  103 + }
  104 + };
  105 +
  106 + }
  107 + var opts = {
  108 + dirname: './logstatalarm/', // required
  109 + log: {
  110 + rotation: 15 * 60 * 1000, // not required, default: 15 minutes
  111 + maxsize: 2000000 // not required, default: 20 Mb
  112 + }, // required ({} if no configuration is needed)
  113 + stat: {
  114 + rotation: 30 * 60 * 1000, // not required, default: 15 minutes
  115 + maxsize: 2000000, // not required, default: 20 Mb
  116 + interval: 2 * 60 * 1000, // not required, default: 1 minute
  117 + data: [
  118 + {key: 'userLogin', threshold_inv: 100},
  119 + {key: 'Error 400', threshold: 1000, threshold_inv: 10},
  120 + {key: 'Error 500', threshold: 1}
  121 + ] // not required, default: []
  122 + }, // required ({} if no configuration is needed)
  123 + alarm: {
  124 + rotation: 60 * 60 * 1000, // not required, default: 15 minutes
  125 + maxsize: 2000000 // not required, default: 20 Mb
  126 + external: [
  127 + { fn: http_request, args: ["http://localhost", "3000", "/alarm", "POST"] }, // required
  128 + { fn: http_request, args: ["http://someserver.com", "9999", "/alarm", "POST"] } // not required
  129 + ] // not required
  130 + }, // required ({} if no configuration is needed)
  131 + };
  132 +
  133 + //Call init method only once
  134 + var logObj = logstatalarm.init(opts);
  135 +
  136 + // NOTE *: in case if there is any error, the module will throw error and crash the applicatio (Configuration error)
  137 + ```
  138 +
  139 + ##### Log
  140 + We will be using variable logObj throughout the application
  141 + ```
  142 + var exampleLogData = {
  143 + path: '/user',
  144 + method: 'GET',
  145 + resquest: {},
  146 + response: {statusCode: 200}
  147 + }
  148 + logObj.log.append(exampleLogData);
  149 + ```
  150 +
  151 + ##### Stat
  152 + After initialization of the module, start method have to be called once inorder to start the interval logging of stat values
  153 + Then freely use increment method any where any time needed
  154 + We will be using variable logObj throughout the application
  155 + ```
  156 + logObj.stat.start(); // call this once
  157 + logObj.stat.increment('Error 500');
  158 + logObj.stat.increment('Error 400');
  159 + logObj.stat.increment(User + 'user login');
  160 + ```
  161 + At any point in time stop method can be called to stop logging interval
  162 + ```
  163 + logObj.stat.stop();
  164 + ```
  165 + ##### Alarm
  166 + External functions will be executed before logging in to alarm file.
  167 + You don't need to do anything with the alarm module. It will be logged automatically from stat module.
  168 +
  169 + ##### File management
  170 + Example of log file with this configuration:
  171 + rotation: 1000
  172 + maxsize: 20000
  173 +
  174 + -rw-r--r-- test 20K Jun 20 15:00 2016-06-20T15:00:19_00001.txt
  175 + -rw-r--r-- test 11K Jun 20 15:00 2016-06-20T15:00:19_00002.txt
  176 + -rw-r--r-- test 20K Jun 20 15:00 2016-06-20T15:00:20_00001.txt
  177 + -rw-r--r-- test 11K Jun 20 15:00 2016-06-20T15:00:20_00002.txt
  178 +
  179 + As you can see, the file is set to maximum at 20Kb and is rotate every 1 second
  180 + If the file reaches maxsize before 1 second it will append log into the same time frame with incremental suffix
  181 +
  182 +
... ...
index.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/index.js
... ... @@ -0,0 +1,60 @@
  1 +var _ = require('lodash');
  2 +var moment = require('moment');
  3 +var log = require('./model/log');
  4 +var stat = require('./model/stat');
  5 +var alarm = require('./model/alarm');
  6 +var validate = require('./lib/validate');
  7 +var joi = require('joi');
  8 +
  9 +module.exports = {
  10 +
  11 + /*
  12 + * opts: {
  13 + * log: [dirname, rotation, maxsize]
  14 + * stat: [dirname, rotation, maxsize, interval, key]
  15 + * alarm: [dirname, rotation, maxsize]
  16 + * }
  17 + */
  18 + init: function(opts) {
  19 + var logstatalarm = undefined;
  20 + validate.joi(opts, this.getSchema(), function(err, opts) {
  21 + if(err) {
  22 + throw new Error(err);
  23 + }
  24 + });
  25 + return { log: new log(opts.dirname, opts.log),
  26 + stat: new stat(opts.dirname, opts.stat, opts.alarm)
  27 + };
  28 + },
  29 +
  30 + /*
  31 + * Pre-define schema
  32 + */
  33 + getSchema: function() {
  34 + return joi.object().keys({
  35 + dirname: joi.string().required(),
  36 + log: joi.object().keys({
  37 + rotation: joi.number().min(0),
  38 + maxsize: joi.number().min(0)
  39 + }).required(),
  40 + stat: joi.object().keys({
  41 + rotation: joi.number().min(0),
  42 + maxsize: joi.number().min(0),
  43 + interval: joi.number().min(0),
  44 + data: joi.array().items(joi.object().keys({
  45 + key: joi.string().required(),
  46 + threshold: joi.number().min(0),
  47 + threshold_inv: joi.number().min(0)
  48 + }))
  49 + }).required(),
  50 + alarm: joi.object().keys({
  51 + rotation: joi.number().min(0),
  52 + maxsize: joi.number().min(0),
  53 + external: joi.array().items(joi.object().keys({
  54 + fn: joi.func().required(),
  55 + args: joi.array().items(joi.any())
  56 + }))
  57 + }).required()
  58 + });
  59 + }
  60 +};
... ...
lib/helper.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/lib/helper.js
... ... @@ -0,0 +1,40 @@
  1 +var fs = require('fs');
  2 +var request = require('request-promise').defaults({jar: true})
  3 +
  4 +module.exports = {
  5 +
  6 + isValidFileSize: function(dirname, maxsize) {
  7 + return fs.statSync(dirname)['size'] < maxsize;
  8 + },
  9 +
  10 + getLengthOfContent: function(string) {
  11 + return Buffer.byteLength(string, 'utf8');
  12 + },
  13 +
  14 + mkdirIfNotExist: function(dirname) {
  15 + var _dirname = dirname.split('/');
  16 + var parentdir = _dirname.slice(0, _dirname.length-2).join('/');
  17 + if (!fs.existsSync(parentdir)){
  18 + fs.mkdirSync(parentdir);
  19 + }
  20 + if (!fs.existsSync(dirname)){
  21 + fs.mkdirSync(dirname);
  22 + }
  23 + },
  24 +
  25 + deleteFolderRecursive: function(path) {
  26 + var self = this;
  27 + if( fs.existsSync(path) ) {
  28 + fs.readdirSync(path).forEach(function(file, index){
  29 + var curPath = path + "/" + file;
  30 + if(fs.lstatSync(curPath).isDirectory()) {
  31 + self.deleteFolderRecursive(curPath);
  32 + } else {
  33 + fs.unlinkSync(curPath);
  34 + }
  35 + });
  36 + fs.rmdirSync(path);
  37 + }
  38 + }
  39 +
  40 +};
... ...
lib/validate.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/lib/validate.js
... ... @@ -0,0 +1,39 @@
  1 +var joi = require('joi');
  2 +var _ = require('lodash');
  3 +
  4 +module.exports = {
  5 +
  6 + joi: function(data, schema, callback) {
  7 + opts = {
  8 + abortEarly: false,
  9 + convert: true,
  10 + allowUnknown: false,
  11 + stripUnknown: false
  12 + }
  13 + joi.validate(data, schema, opts, function(err, obj){
  14 + if(err) {
  15 + var errs = _.map(err.details, function(e) {
  16 + var key = e.path;
  17 + var x = e.type;
  18 + var _postfix = undefined;
  19 + switch(_.last(x.split('.'))) {
  20 + case 'required':
  21 + _postfix = 'required';
  22 + break;
  23 + case 'allowUnknown':
  24 + _postfix = 'not allowed';
  25 + break;
  26 + default:
  27 + _postfix = 'invalid format'
  28 + break;
  29 + }
  30 + return `${key} is ${_postfix}`;
  31 + });
  32 + callback(errs, null);
  33 + } else {
  34 + callback(null, obj);
  35 + }
  36 +
  37 + })
  38 + }
  39 +};
... ...
model/alarm.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/model/alarm.js
... ... @@ -0,0 +1,102 @@
  1 +var fs = require('fs');
  2 +var moment = require('moment');
  3 +var helper = require('../lib/helper');
  4 +var _ = require('lodash');
  5 +
  6 +/*
  7 + * parameters:
  8 + * dirname: string
  9 + * rotation: interger (ms)
  10 + * maxsize: interger (byte)
  11 + */
  12 +function alarm(dirname, opts) {
  13 + opts = opts || {};
  14 + this.dirname = dirname;
  15 + this.rotation = opts.rotation || 15 * 60 * 1000;
  16 + this.maxsize = opts.maxsize || 20000;
  17 + this.external = opts.external || [];
  18 + this.currentsize = 0;
  19 + this.timestamp = 0;
  20 + this.foldername = 'alarm/';
  21 + helper.mkdirIfNotExist(`${this.dirname}/${this.foldername}`);
  22 +};
  23 +
  24 +/*
  25 + * parameters:
  26 + * data: any
  27 + */
  28 +alarm.prototype.appendAlarm = function(data) {
  29 + data = this.formatData(data);
  30 + this.currentsize = this.currentsize + helper.getLengthOfContent(data);
  31 + this.request(data);
  32 + fs.appendFile(this.getDir(), data, function(err) {});
  33 +};
  34 +
  35 +/*
  36 + * parameters:
  37 + * none
  38 + */
  39 +alarm.prototype.getDir = function() {
  40 + var time = moment(Math.floor((+moment()) / this.rotation) * this.rotation);
  41 + this.resetCurrentSize(time.unix());
  42 + time = time.format('YYYY-MM-DDTHH-mm-ss');
  43 + var count = this.getCount();
  44 + return `${this.dirname}${this.foldername}${time}_${count}.txt`;
  45 +};
  46 +
  47 +/*
  48 + * parameters:
  49 + * time_unix: string
  50 + */
  51 +alarm.prototype.resetCurrentSize = function(time_unix) {
  52 + if(time_unix > this.timestamp) {
  53 + this.currentsize = 0
  54 + this.timestamp = time_unix;
  55 + }
  56 +};
  57 +
  58 +/*
  59 + * parameters:
  60 + * none
  61 + */
  62 +alarm.prototype.getCount = function() {
  63 + var count = Math.floor((this.currentsize / this.maxsize) + 1);
  64 + return ((count * 1e-5).toFixed(5)).split('.')[1];
  65 +};
  66 +
  67 +/*
  68 + * parameters:
  69 + * data: any
  70 + */
  71 +alarm.prototype.formatData = function(data) {
  72 + var date = moment().toISOString().trim();
  73 + var timestamp = moment().unix();
  74 + data = this._formatObject(data).trim();
  75 + return `${date} ${timestamp} ${data}\r\n`;
  76 +};
  77 +
  78 +/*
  79 + * parameters:
  80 + * data: any
  81 + */
  82 +alarm.prototype._formatObject = function(data) {
  83 + if(_.isObject(data)) {
  84 + return JSON.stringify(data);
  85 + }
  86 + if(_.isNumber(data)) {
  87 + return toString(data);
  88 + }
  89 + return data;
  90 +};
  91 +
  92 +/*
  93 + * parameters:
  94 + * data: any
  95 + */
  96 +alarm.prototype.request = function(data) {
  97 + _.forEach(this.external, function(external) {
  98 + external.fn.apply(this, [data].concat(external.args));
  99 + });
  100 +};
  101 +
  102 +module.exports = alarm;
... ...
model/log.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/model/log.js
... ... @@ -0,0 +1,89 @@
  1 +var fs = require('fs');
  2 +var moment = require('moment');
  3 +var helper = require('../lib/helper');
  4 +var _ = require('lodash');
  5 +
  6 +/*
  7 + * parameters:
  8 + * dirname: string
  9 + * rotation: interger (ms)
  10 + */
  11 +function log(dirname, opts) {
  12 + opts = opts || {};
  13 + this.dirname = dirname;
  14 + this.rotation = opts.rotation || 15 * 60 * 1000;
  15 + this.maxsize = opts.maxsize || 20000;
  16 + this.currentsize = 0;
  17 + this.timestamp = 0;
  18 + this.foldername = 'log/';
  19 + helper.mkdirIfNotExist(`${this.dirname}/${this.foldername}`);
  20 +};
  21 +
  22 +/*
  23 + * parameters:
  24 + * data: any
  25 + */
  26 +log.prototype.append = function(data) {
  27 + data = this.formatData(data);
  28 + this.currentsize = this.currentsize + helper.getLengthOfContent(data);
  29 + fs.appendFile(this.getDir(), data, function(err) {});
  30 +};
  31 +
  32 +/*
  33 + * parameters:
  34 + * none
  35 + */
  36 +log.prototype.getDir = function() {
  37 + var time = moment(Math.floor((+moment()) / this.rotation) * this.rotation);
  38 + this.resetCurrentSize(time.unix());
  39 + time = time.format('YYYY-MM-DDTHH-mm-ss');
  40 + var count = this.getCount();
  41 + return `${this.dirname}${this.foldername}${time}_${count}.txt`;
  42 +};
  43 +
  44 +/*
  45 + * parameters:
  46 + * time_unix: string
  47 + */
  48 +log.prototype.resetCurrentSize = function(time_unix) {
  49 + if(time_unix > this.timestamp) {
  50 + this.currentsize = 0
  51 + this.timestamp = time_unix;
  52 + }
  53 +};
  54 +
  55 +/*
  56 + * parameters:
  57 + * none
  58 + */
  59 +log.prototype.getCount = function() {
  60 + var count = Math.floor((this.currentsize / this.maxsize) + 1);
  61 + return ((count * 1e-5).toFixed(5)).split('.')[1];
  62 +};
  63 +
  64 +/*
  65 + * parameters:
  66 + * data: any
  67 + */
  68 +log.prototype.formatData = function(data) {
  69 + var date = moment().toISOString().trim();
  70 + var timestamp = moment().unix();
  71 + data = this._formatObject(data).trim();
  72 + return `${date} ${timestamp} ${data}\r\n`;
  73 +};
  74 +
  75 +/*
  76 + * parameters:
  77 + * data: any
  78 + */
  79 +log.prototype._formatObject = function(data) {
  80 + if(_.isObject(data)) {
  81 + return JSON.stringify(data);
  82 + }
  83 + if(_.isNumber(data)) {
  84 + return toString(data);
  85 + }
  86 + return data;
  87 +};
  88 +
  89 +module.exports = log;
... ...
model/stat.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/model/stat.js
... ... @@ -0,0 +1,179 @@
  1 +var fs = require('fs');
  2 +var moment = require('moment');
  3 +var helper = require('../lib/helper');
  4 +var _ = require('lodash');
  5 +var alarm = require('./alarm');
  6 +
  7 +/*
  8 + * parameters:
  9 + * dirname: string
  10 + * rotation: interger (ms)
  11 + * maxsize: interger (byte)
  12 + * interval: interger (ms)
  13 + * data: array (array of object)
  14 + * alarm: object (alarm object)
  15 + */
  16 +function stat(dirname, opts, alarmData) {
  17 + opts = opts || {};
  18 + alarmData = alarmData || {};
  19 + this.dirname = dirname;
  20 + this.rotation = opts.rotation || 15 * 60 * 1000;
  21 + this.maxsize = opts.maxsize || 20000;
  22 + this.currentsize = 0;
  23 + this.timestamp = 0;
  24 + this.foldername = 'stat/';
  25 + this.intervalId = undefined;
  26 + this.interval = opts.interval || 60 * 1000;
  27 + this.data = transformKeys(opts.data);
  28 + this.rules = opts.data;
  29 + this.alarm = new alarm(dirname, alarmData);
  30 + helper.mkdirIfNotExist(`${this.dirname}/${this.foldername}`);
  31 +};
  32 +
  33 +function transformKeys(data) {
  34 + keys = _.map(data, function(obj) {
  35 + var fakey = {};
  36 + fakey[obj.key] = 0;
  37 + return fakey;
  38 + });
  39 + return _.reduce(keys, function(new_obj, obj) {
  40 + return _.merge(new_obj, obj);
  41 + }, {});
  42 +
  43 +};
  44 +
  45 +/*
  46 + * parameters:
  47 + * data: obj
  48 + * rules: [obj]
  49 + */
  50 +function alarmDataOutOfThreshold(data, rules) {
  51 + _alarm = [];
  52 + _.forEach(data, function(v, k) {
  53 + rule = _.find(rules, ['key', k]);
  54 + var alarmObj = {};
  55 + if(v < rule.threshold_inv) {
  56 + _alarm.push({key: k,
  57 + count: v,
  58 + threshold_inv: rule.threshold_inv,
  59 + message: `${k} count is below inverted threshold`});
  60 + }
  61 + if(v > rule.threshold) {
  62 + _alarm.push({key: k,
  63 + count: v,
  64 + threshold:
  65 + rule.threshold,
  66 + message: `${k} count is above threshold`});
  67 + }
  68 + });
  69 + return _alarm
  70 +};
  71 +
  72 +/*
  73 + * parameters:
  74 + * data: any
  75 + */
  76 +stat.prototype.appendStat = function(data) {
  77 + data = this.formatData(data);
  78 + this.currentsize = this.currentsize + helper.getLengthOfContent(data);
  79 + fs.appendFile(this.getDir(), data, function(err) {});
  80 +};
  81 +
  82 +/*
  83 + * parameters:
  84 + * none
  85 + */
  86 +stat.prototype.start = function() {
  87 + var self = this;
  88 + this.intervalId = setInterval(function(){
  89 + var alarmData = alarmDataOutOfThreshold(self.data, self.rules);
  90 + if(!_.isEmpty(alarmData)) {
  91 + self.alarm.appendAlarm(alarmData);
  92 + }
  93 + self.appendStat(self.data);
  94 + self.reset();
  95 + }, self.interval);
  96 +};
  97 +
  98 +/*
  99 + * parameters:
  100 + * none
  101 + */
  102 +stat.prototype.stop = function() {
  103 + clearInterval(this.intervalId);
  104 +};
  105 +
  106 +/*
  107 + * parameters:
  108 + * data: string
  109 + */
  110 +stat.prototype.increment = function(data) {
  111 + if(_.has(this.data, data)) {
  112 + this.data[data]++;
  113 + }
  114 + else this.data[data] = 1;
  115 +};
  116 +
  117 +/*
  118 + * parameters:
  119 + * none
  120 + */
  121 +stat.prototype.reset = function() {
  122 + var self = this;
  123 + _.forEach(this.data, function(v, k) {
  124 + self.data[k] = 0;
  125 + });
  126 +};
  127 +
  128 +/*
  129 + * parameters:
  130 + * none
  131 + */
  132 +stat.prototype.getDir = function() {
  133 + var time = moment(Math.floor((+moment()) / this.rotation) * this.rotation);
  134 + this.resetCurrentSize(time.unix());
  135 + time = time.format('YYYY-MM-DDTHH-mm-ss');
  136 + var count = this.getCount();
  137 + return `${this.dirname}${this.foldername}${time}_${count}.txt`;
  138 +};
  139 +
  140 +/*
  141 + * parameters:
  142 + * time_unix: string
  143 + */
  144 +stat.prototype.resetCurrentSize = function(time_unix) {
  145 + if(time_unix > this.timestamp) {
  146 + this.currentsize = 0
  147 + this.timestamp = time_unix;
  148 + }
  149 +};
  150 +
  151 +/*
  152 + * parameters:
  153 + * none
  154 + */
  155 +stat.prototype.getCount = function() {
  156 + var count = Math.floor((this.currentsize / this.maxsize) + 1);
  157 + return ((count * 1e-5).toFixed(5)).split('.')[1];
  158 +};
  159 +
  160 +/*
  161 + * parameters:
  162 + * data: any
  163 + */
  164 +stat.prototype.formatData = function(data) {
  165 + var date = moment().toISOString().trim();
  166 + var timestamp = moment().unix();
  167 + data = this._formatObject(data).trim();
  168 + return `${date} ${timestamp} ${data}\r\n`;
  169 +};
  170 +
  171 +/*
  172 + * parameters:
  173 + * data: any
  174 + */
  175 +stat.prototype._formatObject = function(data) {
  176 + return JSON.stringify(data);
  177 +};
  178 +
  179 +module.exports = stat;
... ...
package.json 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/package.json
... ... @@ -0,0 +1,19 @@
  1 +{
  2 + "name": "logstatalarm",
  3 + "version": "1.0.0",
  4 + "description": "",
  5 + "main": "index.js",
  6 + "scripts": {
  7 + "test": "mocha"
  8 + },
  9 + "author": "pupuupup",
  10 + "license": "ISC",
  11 + "dependencies": {
  12 + "bluebird": "^3.4.0",
  13 + "chai": "^3.5.0",
  14 + "joi": "^8.4.2",
  15 + "lodash": "^4.13.1",
  16 + "moment": "^2.13.0",
  17 + "request-promise": "^3.0.0"
  18 + }
  19 +}
... ...
test/alarm.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/test/alarm.js
... ... @@ -0,0 +1,54 @@
  1 +require('./setup');
  2 +var alarm = require('../model/alarm');
  3 +var _ = require('lodash');
  4 +var Promise = require('bluebird');
  5 +var fs = require('fs');
  6 +
  7 +describe('Alarm Model', function() {
  8 +
  9 + var testData1, testData2;
  10 + var testAString = "helloworld";
  11 + var aRequestFunction1 = function(data, aString){
  12 + testData1 = data + "," + aString;
  13 + };
  14 + var aRequestFunction2 = function(data, aString1, aString2){
  15 + testData2 = aString1 + "," + data + "," + aString2;
  16 + };
  17 + var validData = {}
  18 + data = {
  19 + rotation: 5000,
  20 + external: [
  21 + { fn: aRequestFunction1 ,args: [testAString] },
  22 + { fn: aRequestFunction2 ,args: [testAString, testAString] }
  23 + ]
  24 + };
  25 + var alarm_object = new alarm('./log_test/', data);
  26 +
  27 + before(function(done) {
  28 + validData = ['lksdjfjklsdlfjll'];
  29 + done();
  30 + });
  31 +
  32 + it('should appendAlarm asynchronously correctly', function(done) {
  33 + Promise.all(_.map(_.times(200, String), function(n) {
  34 + return Promise.resolve(alarm_object.appendAlarm(validData));
  35 + }))
  36 + .then(function() {
  37 + return Promise.delay(1100);
  38 + })
  39 + .then(function() {
  40 + Promise.all(_.map(_.times(200, String), function(n) {
  41 + return Promise.resolve(alarm_object.appendAlarm(validData));
  42 + }))
  43 + })
  44 + .then(function() {
  45 + expect(testData1.split(",")[1]).eql(testAString)
  46 + expect(testData2.split(",")[0]).eql(testAString)
  47 + expect(testData2.split(",")[2]).eql(testAString)
  48 + })
  49 + .then(function() {
  50 + done();
  51 + });
  52 + });
  53 +
  54 +});
... ...
test/index.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/test/index.js
... ... @@ -0,0 +1,134 @@
  1 +require('./setup');
  2 +var index = require('../index');
  3 +var Promise = require('bluebird');
  4 +
  5 +describe('Main index.js', function() {
  6 +
  7 + var validData = {}
  8 + before(function(done) {
  9 + validData = {
  10 + dirname: './log_test/',
  11 + log: {},
  12 + stat: {
  13 + interval: 2,
  14 + data: [
  15 + {key: 'a', threshold: 2, threshold_inv: 1},
  16 + {key: 'b', threshold: 2 }
  17 + ]
  18 + },
  19 + alarm: {}
  20 + }
  21 + done();
  22 + });
  23 +
  24 + it('init function should return true on cb properly', function(done) {
  25 + var object = index.init(validData);
  26 + expect(object).to.be.an('object');
  27 + expect(object).to.have.property('log');
  28 + expect(object).to.have.property('stat');
  29 + done();
  30 + });
  31 +
  32 + it('init function should return err on cb properly (case 1)', function(done) {
  33 + try {
  34 + var object = index.init({log: 'hi'});
  35 + }
  36 + catch (err) {
  37 + var my_err = err.message.split(',');
  38 + expect(my_err).to.include.members(['log is invalid format', 'dirname is required', 'alarm is required']);
  39 + done();
  40 + }
  41 + });
  42 +
  43 + it('init function should return err on cb properly (case 2)', function(done) {
  44 + try {
  45 + var object = index.init({pupu: 'hi'});
  46 + }
  47 + catch (err) {
  48 + var my_err = err.message.split(',');
  49 + expect(my_err).to.include.members(['pupu is not allowed']);
  50 + done();
  51 + }
  52 + });
  53 +
  54 + it('init function should return err on cb properly (case 3)', function(done) {
  55 + try {
  56 + var object = index.init({pupu: 'hi'});
  57 + }
  58 + catch (err) {
  59 + var my_err = err.message.split(',');
  60 + expect(my_err).to.include.members(['log is required', 'stat is required']);
  61 + done();
  62 + }
  63 + });
  64 +
  65 + it('should fail to create stat object if validdata object contain no key with threshold', function(done) {
  66 + var testData = {
  67 + dirname: './log_test/',
  68 + log: {},
  69 + stat: {
  70 + interval: 2,
  71 + data: [
  72 + {key: 'a', threshold: 2, threshold_inv: 1},
  73 + {key: 'b', threshold: 2},
  74 + {key: 'c', threshold_inv: 1},
  75 + {key: 'd', threshold: 2},
  76 + {threshold: 2},
  77 + ]
  78 + }
  79 + }
  80 + try {
  81 + var object = index.init(testData);
  82 + }
  83 + catch (err) {
  84 + var my_err = err.message.split(',');
  85 + expect(my_err).to.include.members(['stat.data.4.key is required']);
  86 + done();
  87 + }
  88 + });
  89 +
  90 + it('should run the whole flow correctly', function(done) {
  91 + var initData = {
  92 + dirname: './whole_flow/',
  93 + log: {
  94 + rotation: 500,
  95 + maxsize: 500
  96 + },
  97 + stat: {
  98 + rotation: 500,
  99 + maxsize: 500,
  100 + interval: 5,
  101 + data: [
  102 + {key: 'testIncrement1', threshold: 3, threshold_inv: 1},
  103 + {key: 'testIncrement2', threshold: 2 }
  104 + ]
  105 + },
  106 + alarm: {
  107 + rotation: 500,
  108 + maxsize: 500
  109 + }
  110 + };
  111 + var logObj = index.init(initData);
  112 + Promise.try(function() {})
  113 + .then(function() {
  114 + logObj.stat.start();
  115 + logObj.log.append({hi: 'test'});
  116 + logObj.stat.increment('testIncrement1');
  117 + return Promise.delay(100);
  118 + })
  119 + .then(function() {
  120 + logObj.log.append({hi: 'test2'});
  121 + logObj.stat.increment('testIncrement2');
  122 + logObj.log.append({hi: 'test2'});
  123 + logObj.stat.increment('testIncrement2');
  124 + })
  125 + .then(function() {
  126 + logObj.stat.stop();
  127 + done();
  128 + })
  129 +
  130 +
  131 + });
  132 +
  133 +
  134 +});
... ...
test/lib/helper.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/test/lib/helper.js
... ... @@ -0,0 +1,26 @@
  1 +require('../setup');
  2 +var helper = require('../../lib/helper');
  3 +
  4 +describe('Lib helper.js', function() {
  5 +
  6 + var _dirname = './test/mocha.opts';
  7 + before(function(done) {
  8 + done();
  9 + });
  10 +
  11 + it('isValidFileSize should return true when file size is less than maxsize', function(done) {
  12 + var _maxsize = 200;
  13 + bool = helper.isValidFileSize(_dirname, _maxsize);
  14 + expect(bool).to.be.true;
  15 + done();
  16 + });
  17 +
  18 + it('isValidFileSize should return false when file size exceed max size', function(done) {
  19 + var _maxsize = 10;
  20 + bool = helper.isValidFileSize(_dirname, _maxsize);
  21 + expect(bool).to.be.false;
  22 + done();
  23 + });
  24 +
  25 +
  26 +});
... ...
test/log.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/test/log.js
... ... @@ -0,0 +1,62 @@
  1 +require('./setup');
  2 +var log = require('../model/log');
  3 +var _ = require('lodash');
  4 +var Promise = require('bluebird');
  5 +var fs = require('fs');
  6 +
  7 +describe('Log Model', function() {
  8 +
  9 + var validData = {}
  10 + var log_object = new log('./log_test/', {rotation: 1000, maxsize: 20000});
  11 +
  12 + before(function(done) {
  13 + validData = {
  14 + path: "/test",
  15 + method: "GET",
  16 + request: {
  17 + body: {test: true}
  18 + },
  19 + response: {
  20 + statusCode: 200,
  21 + body: { success: true }
  22 + }
  23 + };
  24 + done();
  25 + });
  26 +
  27 + it('should formatObject correctly', function(done) {
  28 + var data = log_object._formatObject(validData);
  29 + expect(data).eql(JSON.stringify(validData));
  30 + done();
  31 + });
  32 +
  33 + it('should formatData correctly', function(done) {
  34 + var data = log_object.formatData(validData).split(' ');
  35 + expect(_.last(data).slice(0, _.last(data).length-1).trim().split('\r\n').join(''))
  36 + .eql(JSON.stringify(validData).trim().split('\r\n').join(''));
  37 + done();
  38 + });
  39 +
  40 + it('should appendLog correctly', function(done) {
  41 + log_object.append(validData);
  42 + done();
  43 + });
  44 +
  45 + it('should appendLog asynchronously correctly', function(done) {
  46 + Promise.all(_.map(_.times(200, String), function(n) {
  47 + return Promise.resolve(log_object.append(validData));
  48 + }))
  49 + .then(function() {
  50 + return Promise.delay(1100);
  51 + })
  52 + .then(function() {
  53 + Promise.all(_.map(_.times(200, String), function(n) {
  54 + return Promise.resolve(log_object.append(validData));
  55 + }))
  56 + })
  57 + .then(function() {
  58 + done();
  59 + });
  60 + });
  61 +
  62 +});
... ...
test/mocha.opts 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/test/mocha.opts
... ... @@ -0,0 +1,4 @@
  1 +--recursive
  2 +--reporter spec
  3 +--slow 500
  4 +--timeout 3000
... ...
test/setup.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/test/setup.js
... ... @@ -0,0 +1,8 @@
  1 +global.expect = require('chai').expect;
  2 +var fs = require('fs');
  3 +var helper = require('../lib/helper');
  4 +
  5 +before(function (done) {
  6 + helper.deleteFolderRecursive('./log_test/');
  7 + done();
  8 +});
... ...
test/stat.js 0 → 100644
  Diff comments   View file @adfda8e
  1 +++ a/test/stat.js
... ... @@ -0,0 +1,49 @@
  1 +require('./setup');
  2 +var stat = require('../model/stat');
  3 +var _ = require('lodash');
  4 +var Promise = require('bluebird');
  5 +var fs = require('fs');
  6 +
  7 +describe('Stat Model', function() {
  8 +
  9 + var validData = [
  10 + {key: 'a', threshold: 2, threshold_inv: 1},
  11 + {key: 'b', threshold: 2},
  12 + {key: 'c', threshold_inv: 1},
  13 + {key: 'd', threshold: 2}
  14 + ]
  15 +
  16 + var stat_object = new stat('./log_test/', {rotation:5000, maxsize:5000, interval:1, data:validData})
  17 +
  18 + before(function(done) {
  19 + validData = ['a', 'b'];
  20 + done();
  21 + });
  22 +
  23 + it('should stat correctly (whole process)', function(done) {
  24 + stat_object.start();
  25 + return Promise.delay(100)
  26 + .then(function() {
  27 + Promise.all(_.map(_.times(100, String), function(){
  28 + stat_object.increment(validData[0]);
  29 + }))
  30 + })
  31 + .then(function() {
  32 + return Promise.delay(100)
  33 + })
  34 + .then(function() {
  35 + stat_object.increment(validData[0]);
  36 + stat_object.increment(validData[1]);
  37 + })
  38 + .then(function() {
  39 + return Promise.delay(100)
  40 + })
  41 + .then(function() {
  42 + stat_object.stop();
  43 + done();
  44 + })
  45 +
  46 + });
  47 +
  48 +
  49 +});
... ...