Logging service

Owned by Vladimir Tayzhev
Last updated: Apr 01, 2021 by Technical Service Specialist
15 min read

Article contents:


Logging service is used to record the events occurring in the system in time.

Logging is possible only if you have the "log" permission in the profile of the user who is performing the request.

To control the logs, use the following values ​​of the querytype field:

1) log - get the log. Extra options are:

  • time - output the log of events that occurred later than specified (Unix Time Stamp format - UTS);
  • offset - output the event log, omitting the offset events, starting with the most recent event (ie, the event with the maximum id), is used for page-by-page reading;
  • count - output no more than count events in the log, used for page-by-page reading;

General attributes for the log entry node:

  • id - the identifier of the event;
  • time - time in UTS format;
  • severity - event weight (importance), possible values ​​(according to RFC 5424):
    • Emergency - the system is not operational (hardware error, currently no software is generated);
    • Alert - urgent intervention is necessary (program error, should be generated when the WatchDog verification script is triggered and when rebooted);
    • Critical - critical events (transition of sensors to Alarm, Low Alarm, High Alarm states; errors that can lead to system failure - for example, lack of available resources in the system and etc.);
    • Error - error messages (errors when performing actions in the system);
    • Warning - warnings (transition of sensors to the Warning, Low Warning, High Warning states; system warnings - ...);
    • Notice - important notifications (renaming of elemets, changing the state of logical schemes, sending notifications, ...);
    • Informational - information messages (???);
    • Debug - debugging messages;
  • type - type of event, possible types of events (???):
    • "state" - change in the state of the element;
    • "rename" - change in the name of the element;
    • "error" - an error;
    • "message" - a message;
    • "firmware update" - message of firmware update service;
    • "firmware update error" - error while working with the firmware update service;
    • "send mail" - the message of E-mail service;
    • "send mail error" - error while sending e-mail notification;
    • "send sms" - SMS notification service message;
    • "send sms error" - error while sending SMS notification service;
    • "access" - access error;
    • "httpd info" - web server messages;

Types of events "state" and "rename" also contain the attributes of the element for which the event occurred:

  • eid - id of the element;
  • element - the type of the element (??? change to etype, add eclas or even remove it with name ???);
  • name - the name of the element (??? eid - enough ???);
  • old - previous name or state;
  • new - new name or state;

The other types of events contain only the attribute:

  • message - text description of the event;


Example
<log>
  <entry id="61" time="1399898505" severity="Natice" type="message" message="Logic &apos;Температура&apos;(id=3) was activated" />
  <entry id="60" time="1399898505" severity="Warning" type="state" eid="1001" element="temperature" name="Internal T°C" old="high alarm" new="high warning" />
  <entry id="59" time="1399898406" severity="Critical" type="state" eid="1054" element="temperature" name="sc450-0020-temperature" old="low alarm" new="high alarm" />
  <entry id="58" time="1399898402" severity="Critical" type="state" eid="1020" element="temperature" name="sc470-0003-internal T°C" old="low alarm" new="high alarm" />
  <entry id="57" time="1399898369" severity="Notice" type="state" eid="1056" element="motion" name="sc408-2147483647-analog-4" old="unknown" new="normal" />
  <entry id="56" time="1399898365" severity="Critical" type="state" eid="1004" element="GSM modem" name="GSM Modem" old="normal" new="alarm" />
</log>

2) logdate - get the time of the last event in the log and the current time;

The attributes of the logdate node are:

  • time - the time of the last event in the system (UTS);
  • current - the current time in the system (UTS);
  • total - the number of events in the log;
  • maxid - id of the last event in the log;


Example
<logdate time="1355137672" current="1355138383" total="6" maxid="6" />

3) clearlog - clear log;

The current log is unloaded by calling the cgi script rss.htm ("GET" method). Additional parameters are:

  • save - Value:
    • 0 - in the format of the RSS document;
    • 1 - in format for saving to disk;

SD card operation

The SD card is used for recording the changes in the system: logs, sensor readings, element settings.

Supported format is FAT32 and Ext4 (preferably). The SD card must be formatted by the user. The contents of the SD card can be obtained at the address /sdcard/, with login and password hash of current user, for example:

http://{login}:{hash}@192.168.0.193/sdcard/

The files are recorded in a directory with the name formed from the device type name (for example VT300) and the value of the Network→Hostname field, and has the following form: VT300-DeviceName. If the name is not specified by the user (by default is: hostname), then the IP address of the device is used to form the directory name, and the name has the following form: VT300-192.168.0.193. Within this directory, directories are created for log files (log), dump files of changes (diff), and full dump files (full). File names are formed based on the time of their creation, and have the following form: 2016.11.17_16-37-43.log. The file name extension reflects its contents: log - for log files, diff.csv - for dump files of changes, full.dump - for full dump files. At the beginning of the log file, when it is created, service information about the type of device, firmware version, IP address and etc. Log files are written in text form. Dump files of changes (diffs) and dumps of sensor values ​​are saved in text table format CSV. Full dump files are saved in text form.

Video files are saved in the subdirectory ./video/ , with the index of the camera with which the clip was received. The video format is AVI, the name is formed the same way as for the other files.


The log file is filled in as events appear, and looks like:

Example
#
# Device type: "VT335", Firmware version: "2.7.1".
# Hostname: "hostname", IP: "192.168.0.199", MAC: "unknown"
#
Thu Nov 17 14:23:20 2016    message: USB Flash was inserted
Thu Nov 17 14:23:25 2016    state: Element Onboard Temperature (id=201001) changed state from "unknown" to "normal" 
Thu Nov 17 14:23:25 2016    state: Element Onboard Voltage DC (id=203001) changed state from "unknown" to "normal" 
Thu Nov 17 14:23:26 2016    state: Element Dry-1 (id=101001) changed state from "unknown" to "normal" 
Thu Nov 17 14:23:26 2016    state: Element Dry-2 (id=101002) changed state from "unknown" to "normal" 
Thu Nov 17 14:23:30 2016    message: System started up and ready
Thu Nov 17 14:23:41 2016    state: Element GSM Modem (id=404001) changed state from "unknown" to "normal"

The dump file of changes is populated periodically (every 60 seconds), and reflects the identifiers and values of the sensors whose values have changed over the elapsed time period. Every 60 minutes, the dump records the values of all sensors, even if their values have not changed. Dump changes in text form:

Example
Device type: "VT335" 
Firmware version: "2.7.1" 
Hostname: "spirt" 
IP: "192.168.0.193" 
MAC: "d8:80:39:9e:97:a3" 
Date and Time 
Thu May 11 16:45:54 2017,201001,201002 
,24.100,22.600 
Thu May 11 16:46:54 2017,201001 
,24.000

The full dump file is filled periodically (every 60 minutes), and contains a complete description of the elements and modules of the system:

Example
Thu Nov 17 14:28:52 2016
    [101001]={module:"2022";clas:"discrete";type:"dry";name:"Dry-1";state:"normal";hwport:"1";value:"0";utype:"";reverse:"0";};
    [101002]={module:"2022";clas:"discrete";type:"dry";name:"Dry-2";state:"normal";hwport:"2";value:"0";utype:"";reverse:"0";};
    [203001]={module:"2020";clas:"analog";type:"voltage";name:"Onboard Voltage DC";state:"normal";value:"12.2";vmax:"24.0";vmin:"0.0";low_alarm:"9.0";low_warning:"11.0";high_warning:"13.0";high_alarm:"14.0";at0:"1.000";at75:"0.000";um:"V";};
    [301001]={module:"2021";clas:"switch";type:"relay";name:"Analog Power";state:"on";value:"1";initial:"on";pulse:"15";};
    [304001]={module:"2023";clas:"switch";type:"strobo";name:"Power-1";state:"off";value:"0";initial:"off";pulse:"2";hwport:"1";};
    [403001]={module:"2031";num:"1";clas:"devhwr";type:"usb-cam";name:"USB Web camera";url:"";fps:"1.0";username:"";password:"";resolution:"320x240";};
    [404001]={module:"2030";clas:"devhwr";type:"GSM modem";state:"normal";name:"GSM Modem";value:"90";imei:"866104023288015";pin:"";sms_centre:"+79168999100";allowed:"";status:"found, ready";verbosity:"2";};

The user can set the following logging parameters through the API :

  • Enable or disable the logging feature on the media;
  • lifetime of files - all older files are deleted from the storage media (7 days by default).

When the system is rebooted, new files are created for writing. The same happens when installing a new SD card. Required directories are created automatically if logging is enabled.

The following values ​​of the querytype field are used for management:

  • getsdcard - get information about the map, returns the response in the form

    <sdcard state="normal" mpoint="/media/sdcard-sd54a1" fullsize="7808156" freesize="7806524" lifetime="31536000" enablelog="1" enablelogvideo="1" />

    , where:

    • state - the state of the medium:
      • normal - the operating state, the carrier is installed;
      • not connected - the medium is not installed or removed from the connector;
      • off - the media is unmounted with the umount command;
    • fullsize - total amount of memory, in KB;
    • freesize - the amount of free memory in KB;
    • lifetime - the lifetime of the files, in seconds (the default is 7 days, ie 604800 seconds);
    • enablelog - logging to this media is enabled or disabled  (0 - disabled, 1 - enabled);
    • enablelogvideo - saving of video files to this media is enable or disable  (0 - disabled, 1 - enabled);
  • setsdcard - set the logging parameters on the SD card. Extra options are:
    • cmd - command; it's possible values are:
      • umount - unmount the file system for safe extraction;
      • restart - restart, close current log and dump files and create new files;
    • lifetime - the lifetime of the files, in seconds (minimum 3 hours, maximum 1 year);
    • enablelog - enable / disable the logging process to this media (0 - disabled, 1 - enabled);
    • enablelogvideo - enable / disable saving of video to this media (0 - disabled, 1 - enabled);

When specifying parameters, if the new configuration is different from the current one, the current log and dump files are closed, new files are created and new logging parameters are applied. In addition, the settings are saved in the device's permanent memory, an additional save command is not required.

Setting the lifetime, and enablelog parameters is possible in the absence of the installed media. "Umount" command will return an error message when there is no media. "Restart" command can also be executed in the absence of media and if the module is operational, a successful response will be returned, but no action will be taken.

When you remove the SD card without first unmounting, data loss is possible.

Working with USB Flash Drive

The USB flash drive is used to record changes in the system: logs, sensor readings, element settings.

The FAT32 file system is supported. USB Flash Drive must be formatted by the user. The contents of the USB Flash Drive can be obtained from the address /usbflash/, with login and password hash of current user, for example:

http://{login}:{hash}@192.168.0.193/usbflash/


The names of directories and files are also formed as when working with SD Card.

The user can set the following logging parameters through the API:

  • enable and disable the logging function;
  • lifetime of files - all older files are deleted from the USB Flash Drive memory (by default 7 days).

When the system is rebooted, new files are created for writing. The same happens when installing a new USB flash drive. Required directories are created automatically if logging is enabled.

The following values of the querytype field are used for management:

  • getusbflash - get information about the flash drive, returns the response as 

    <usbflash state="normal" mpoint="/media/usbhd-sda1" fullsize="7808156" freesize="7806524" lifetime="31536000" enablelog="1" enablelogvideo="1" />

    , where:

    • state - USB Flash Drive status:
      • normal - working state, the flash drive is installed;
      • not connected - the flash drive is not installed or removed from the connector;
      • off - the flash drive is unmounted by the umount command;
    • fullsize - total amount of memory, in KB;
    • freesize - the amount of free memory in KB;
    • lifetime - the lifetime of the files, in seconds (the default is 7 days, ie 604800 seconds);
    • enablelog - logging to flash is enabled or disabled  (0 - disabled, 1 - enabled);
    • enablelogvideo - saving of video files to this media is enabled or disabled (0 - disabled, 1 - enabled);
  • setusbflash - set the parameters of the log on the USB flash drive. Extra options are:
    • cmd - command; it's  possible values are:
      • umount - unmount the file system for safe extraction;
      • restart - restart, close current log and dump files and create new files;
    • lifetime - the lifetime of the files, in seconds (minimum 3 hours, maximum 1 year);
    • enablelog - logging process on the USB flash drive is enabled or disabled (0 - disabled, 1 - enabled);
    • enablelogvideo - saving video to USB flash drive enabled or disabled (0 - disabled, 1 - enabled).

When specifying parameters, if the new configuration is different from the current one, the current log and dump files are closed, new files are created and new logging parameters are applied. In addition, the settings are saved in the device's permanent memory and additional save command is not required.

Setting the parameters period, maxfiles, lifetime and enablelog is possible in the absence of the installed USB flash drive. The command umount in the absence of a flash drive will return an error message. "Restart" command can also be executed in the absence of a flash drive and if the module is operational, a successful response will be returned, but no action will be taken.

When you remove the USB flash drive without first unmounting, data loss is possible.

Working with SATA Hard Disk Drive

The SATA HDD is used to record changes in the system: logs, sensor readings, element settings, video records.

The FAT32 and Ext4 file system is supported. HDD must be formatted by the user. The contents of the HDD can be obtained from the address /satahdd/, with login and password hash of current user, for example:

http://{login}:{hash}@192.168.0.193/satahdd/


The names of directories and files are also formed as when working with SD Card.

The user can set the following logging parameters through the API:

  • enable and disable the logging function;
  • lifetime of files - all older files are deleted from the HDD memory (by default 7 days).

When the system is rebooted, new files are created for writing. Required directories are created automatically if logging is enabled.

The following values of the querytype field are used for management:

  • getsatahdd - get information about the drive, returns the response as 

    <satahdd state="normal" mpoint="/tmp/mnt/satahdd" fullsize="7808156" freesize="7806524" lifetime="31536000" enablelog="1" enablelogvideo="1" />

    , where:

    • state - HDD status:
      • normal - working state, the drive is installed;
      • not connected - the drive is not installed or removed from the connector;
      • off - the drive is unmounted by the umount command;
    • fullsize - total amount of memory, in MB;
    • freesize - the amount of free memory in MB;
    • lifetime - the lifetime of the files, in seconds (the default is 7 days, ie 604800 seconds);
    • enablelog - logging to disk is enabled or disabled  (0 - disabled, 1 - enabled);
    • enablelogvideo - saving of video files to this media is enabled or disabled (0 - disabled, 1 - enabled);
  • setsatahdd - set the parameters of the log on the drive. Extra options are:
    • lifetime - the lifetime of the files, in seconds (minimum 3 hours, maximum 1 year);
    • enablelog - logging process on the drive is enabled or disabled (0 - disabled, 1 - enabled);
    • enablelogvideo - saving video to drive enabled or disabled (0 - disabled, 1 - enabled).

When specifying parameters, if the new configuration is different from the current one, the current log and dump files are closed, new files are created and new logging parameters are applied. In addition, the settings are saved in the device's permanent memory and additional save command is not required.

Setting the parameters lifetime, enablelog and enablelogvideo is possible in the absence of the installed drive. If there is no hardware support for SATA drive, then "not supported" error is returned.

Save the log to the FTP server

Logging and dumping to the FTP server is done through a files on one of the media (in priority order - SATA HDD, SD card, USB flash drive). At a given time (00:00), the data files from the media is transferred to the FTP server. The structure of the folders and files on the remote server is similar to that created on the SD Card and USB Flash Drive.

The user can set the following logging parameters through the API :

  • enable and disable the logging function;
  • lifetime of files - all older files are deleted from the FTP server (by default 7 days);
  • pecify the path to the remote FTP server, its port, and the login and password for it.

The following values ​​of the querytype field are used for management:

  • getftplog - get information about the server for logging; returns a response in the form 

    <ftplog server="ftp://server.com" login="myuser1" port="" enablelog="0" lifetime="110000" period="3600" status="Disable"/>

    , where:

    • server - the address of the FTP server;
    • port - port of the server, if the field is empty, then passive connection mode is used, when parameters are detected automatically;
    • login - user login;
    • password - the password of the user;
    • enablelog - enabled or disabled logging to the server (0 - disabled, 1 - enabled);
    • enablelogvideo - saving of video files to the server iis enabled or disabled (0 - disabled, 1 - enabled);
    • lifetime - the lifetime of the files, in seconds (the default is 7 days, ie 604800 seconds), if the value is 0, the files from the server are not deleted;
    • status - describes the status of the connection to the server:
      • Disable - Logging is disabled;
      • Enable - the logging is enabled, but the connection to the server has not yet been established (there is no data to send to the server yet);
      • Normal - normal operation;
      • messages are reported here if there are connection errors;
  • setftplog - set the logging parameters to the server. Extra options are:
    • server - the address of the FTP server;
    • port - server port, if you specify an empty field, then passive connection mode is used, when parameters are detected automatically;
    • login - user login;
    • password - the password to the server;
    • lifetime - the lifetime of the files, in seconds (minimum 24 hours, a maximum of six months), if the value is 0, files from the server are not deleted;
    • enablelog - logging process to the server is enabled or disabled  (0 - disabled, 1 - enabled);
    • enablelogvideo - saving video to the server is enabled or disabled  (0 - disabled, 1 - enabled).
  • sendftplog - send files (if available) to FTP server with the parameters specified by the "setftplog" command.

When specifying parameters, if the new configuration is different from the current one, the temporary log and dump files are sent to the FTP server, new logging parameters are applied. In addition, the settings are saved in the device's permanent memory, an additional save command is not required.

Sending the log archive file to e-mail

Log files, sensor dumps and etc. at a given time (00:00) compressed into the archive file (*.gz extension) and sent to specified mail addresses. The folder structure in the created archive is similar to that created on the SD Card and USB Flash Drive.

Video files are not archived, but sent to the mail immediately.

The user can set the following logging parameters through the API :

  • enable and disable logging to mail;
  • set the frequency of sending the archive (from 2 to 24 hours);
  • specify the parameters of the outgoing SMTP server;
  • specify the list of recipients to whom the letter with the archive will be sent (up to 10 addresses);
  • arbitrarily send a letter from the specified server.

The following values ​​of the querytype field are used for management:

  • getmaillog - get information about e-mail, returns the response in the form 

    <maillog enablelog="1" sendperiod="7200" server="smtp.my.com" port="465" login="xlam@my.com" password="PAROL5926" enabletls="1" starttls="0" from="xlam@my.com" to="nerd@gmail.com;staff@gmail.com" status="Normal"/>

    , where:

    • enablelog - sending the log archive by mail is enabled or disabled (0 - disabled, 1 - enabled);
    • enablelogvideo - sending video files to the mail is enabled or disabled (0 - disabled, 1 - enabled);
    • server - the address of the SMTP server of outgoing e-mail;
    • port - server port;
    • login - user login (may be empty);
    • password - user's password (may be empty);
    • enabletls - when the TLS SMTP permission is enabled, the session is opened inside the encrypted connection (typical port 465);
    • starttls - when STARTTLS is enabled, the encrypted connection is opened inside the SMTP session (typical port 587, but may be 25, 2525 or another);
    • from - address of the sender;
    • to - addresses of recipients, with a list of up to 10 pieces separated by commas, semicolons or spaces (as the user entered);
    • status - describes the status of the connection to the server:
      • Disable - the function is disabled;
      • Enable - the function is enabled, but communication with the server is not yet installed (there is no data to send to the server);
      • Normal - normal operation;
      • Any connection errors are reported here.
  • setmaillog - set sending options to mail. Extra options are:
    • enablelog - enable or disable the sending of the log archive by mail (0 - disable, 1 - enable);
    • enablelogvideo - enable or disable the sending of video by mail (0 - disable, 1 - enable);
    • server - the address of the SMTP server of outgoing e-mail;
    • port - server port;
    • login - user login (may be empty);
    • password - user's password (may be empty);
    • enabletls - when the TLS SMTP permission is enabled, the session is opened inside the encrypted connection (typical port 465);
    • starttls - when STARTTLS is enabled, the encrypted connection is opened inside the SMTP session (typical port 587, but may be 25, 2525 or another);
    • from - address of the sender;
    • to - addresses of recipients, with a list of up to 10 pieces separated by commas, semicolons or spaces.
  • sendmaillog - send an email with the parameters specified by the "setmaillog" command.

When specifying parameters, if the new configuration is different from the current one, the new parameters are applied and stored in the device's permanent memory and an additional save command is not required.

Per second dump of sensor values

It is possible to record on the specified medium of second-by-second samples of analog and discrete sensors. The data is recorded according to the specified media (HDD, USB-Flash Drive, SD Card). In this case, detailed logging should be enabled (allowed) on the specified medium.

To write files in the root directory, which also contains the log, video and other directories, a directory called sensors is created, inside which directories are created with the name corresponding to the identifier of the observed sensor (201001, 101004, etc.). File names are formed based on the time of their creation, and have the form 2016.11.17_16-37-43.201001.csv. The file name extension includes the identifier of the sensor. The files are written in the text table format CSV.

The per-second dump file is filled in periodically (a group of samples every 30 seconds), and reflects the counting time (in readable form and in UTS format) and the value of the sensor (in a float format with an accuracy of 3 decimal places). For analog sensors, the scores are per second. For discrete, in the absence of state changes, counts can go less often. If there is no sensor, the directory and file are not created and are not filled. Dump in text form:

Example
Device type: "VT335" 
Firmware version: "2.7.1" 
Hostname: "spirt" 
IP: "192.168.0.193" 
MAC: "d8:80:39:9e:97:a3" 
Date and Time,UTS,Value 
Thu May 11 12:23:51 2017,1494505431,23.400 
Thu May 11 12:23:52 2017,1494505432,23.500 
Thu May 11 12:23:53 2017,1494505433,23.600 
Thu May 11 12:23:54 2017,1494505434,23.700


The user can set the following parameters through the API:

  • enable and disable the function of a dump on each of the media for every second;
  • set the list of observed sensors, now up to 10 pieces.

The following values ​​of the querytype field are used for management:

  • getsensdump - get information about sensor dump for every second, returns a response in the form <sensdump elementslist="201001,201002,101001," enable_usb="1" enable_sd="0" enable_ftp="0" enable_mail="1" />, where:
    • elementslist - the list of observed sensors, separated by commas;
    • samplerate - period of sensor data polling, in seconds;
    • enable_hdd - flag to enable the SATA HDD;
    • enable_usb - flag to enable the USB flash drive;
    • enable_sd - the flag to enable the use of SD Card media;
    • enable_ftp - flag to enable use as an FTP server media;
    • enable_mail - the flag to enable sending the log file to e-mail.
  • setsensdump - set the parameters of a sensor dump for every second. Extra options are:
    • elementslist - the list of observed sensors, separated by commas;
    • samplerate - period of sensor data polling, in seconds (1-600);
    • enable_hdd - flag to enable the SATA HDD;
    • enable_usb - flag to enable the USB flash drive;
    • enable_sd - the flag to enable the use of SD Card media;
    • enable_ftp - flag to enable use as an FTP server media;
    • enable_mail - the flag to enable sending of a log file to an email

When specifying parameters, if the new configuration is different from the current one, new logging parameters are applied and parameters are saved in the device's permanent memory; an additional save command is not required.