DLOG_STORE_FILE_CSV append

[laruso]

First, thanks so much for distributing the OSCAT libraries!

I want to use the function block "DLOG_STORE_FILE_CSV" to log data into one file only. Because of the restrictions of retaining variables (for BeagleBone Black the proper shutdown is needed), I want to do some workaround in that way as the data has to be append only to the file.
I know this feature already exists, but to work properly the data in the variable "save_data : DLOG_SAVE" (used as a paramter for "DLOG_STORE_FILE_CSV" instance, has to be a retained variable. Therefore it is necessary to use a UPS, to savely retain variable on my BeagleBone Black. To workaround this, I would like to use an "append" parameter, to not need the variable to be retained.

If I use the code and the demo from the library (on my BeagleBone Black without retaining the retain variables), the content of the file will be erased, due to the empty variable "save_data" of data type "DLOG_SAVE" used as parameter for instance call of "DLOG_STORE_FILE_CSV". For me the functionality of writing with append, should not depend on retaining any variables.

(Btw. I know that that hasn't already be written to file will be lost due to power lost; but this doesn't matter. Morte important is the fact that there is no unlimited storage on devices, so it would be best for me to define a maximum number of "FILE_SIZE" or "line count", to delete the latest log entry and write the new log entry, if the file hits this maximum file size. So there will be only one file with the latest log entries, depending on the defined file size.)

At least the question is how to change the behavior of writing to file (append or delete content and write data) if there is already a file with that filename?

I've read a lot about this function block in this forum. Several times there is the hint to use the append mode, but don't know how to do. Can anyone please give me an advise in how to code that?

In the documentation for the function block "FILE_SERVER" there is also a hint:

An automatic append function can be realized very easy. The parameter FILE_SIZE has to be written to the OFFSET

But I don't know how to implement. When I tried to do it outside the function block instance call, an error occurs because FILE_SIZE is no input of FILE_SERVER. Here is the code I tried:

Code Auswählen Erweitern


x1(X:=x); // Logger input for STRING, used for Time stamp
x2(X:=x, STR:=Glob_Var.sLogExportText2); // Logger input for STRING, used for Event type
x3(X:=x, STR:=Glob_Var.sLogExportText3); // Logger input for STRING, used for Description
DLOG_STORE_FILE_CSV(X:=x, SAVE_DATA:=save_data, ENABLE:=enable, TRIG_M:=Glob_Var.xLogExportTrig_m, FILENAME:=filename, DTI:=RTC_2.UDT, SEP:=9, AUTO_CLOSE:=TIME#10S, ERROR_C=>error_c, ERROR_T=>error_t); // File storage (csv)

RTrig(CLK:=DLOG_STORE_FILE_TSV.FSD.FILE_OPEN);

IF RTrig.Q THEN
DLOG_STORE_FILE_CSV.FSD.OFFSET := DLOG_STORE_FILE_CSV.FSD.FILE_SIZE;
END_IF


When I try to use a copy of the function block "DLOG_STORE_FILE_CSV" from the library everything is OK, but when I try to use a copy of the function block "FILE_SERVER" from the library and to the change there, it gets me several errors (32) when I want to translate the project. Here is the code I tried:

Code Auswählen Erweitern


150: (* File Open *)
IF command = BYTE#0 THEN
IF error THEN
step := 30000; (* Beenden *)
ELSE
(* Filedaten eintragen *)
handle := open_handle;
used_filename := FSD.FILENAME;
file_position := UDINT#4294967295; (* maximal = unbekannt *)
FSD.FILE_OPEN := TRUE;
FSD.OFFSET := FSD.FILE_SIZE;
step := 200;
END_IF;
END_IF;



Till now my workaround was to change the code of "DLOG_STORE_FILE_CSV" like this:

Code Auswählen Erweitern


10: X.UCB.D_STRING := fn_last;
// IF fn_last = SAVE_DATA.FN_REM THEN
X.UCB.D_HEAD := WORD#16#F201; (* Filemode open + write *)
X.ADD_COM := 00;
// ELSE
// X.UCB.D_HEAD := WORD#16#F301; (* Filemode create + write *)
// X.ADD_COM := 02; (* ADD HEADER *)
// END_IF;

X.UCB.D_MODE := 1; (* Dateinamen als String ablegen *)
UCB(DATA:=X.UCB); (* Daten eintragen *)
step_1 := 30;

But this results in losing the opertunity to get a new file, by changing the variable "filename", and this is no goal.


I use CODESYS 3.5.16 Patch 4 and the OSCAT NETWORK library (v1.3.5.2) from CODESYS store.

I'm over this topic for a long while now and pretty looking forward for some help to get it work.

[peewit]

the problem with the limited storage space can be avoided by using several different but always recurring file names.
this way after a certain amount of files the old ones will be overwritten again and again

documentation dlog_store_csv module

With parameters FILENAME the fle name (including path if necessary) is
defned. If the flename is changed during the recprding, it will
automatically on-the-fy changed to the new record fle (with no data loss).
This change can also be automated. The parameter FILE NAME supports
the use of date / time parameter (see documentation from the module
DT_TO_STRF)
Example: FILE NAME = 'Station_01_#R.csv'
At position of '#R' automatically the current minute number is entered. This means
that automatically every minute the file name changes, and therefore the data is
written into the file. Thus, within an entire hour 60 files are created and filled with
data, and in the ring buffer manner overwritten again and again.
53 Version 1.21
Chapter 7. Data Logger
A recording can be done automatically and creates every day, week, month, etc. a
new file as desired. If a new FILE NAME is detected, a possibly existing file is erased
and rewritten.

[laruso]

I really think you didn't get my issue, because I don't have problems with limited storage space. Albeit a limited storage problem can't be fixed with saving data in several files instead of one.

My issue is, that I only want to append the data if the file already exists, without the need of the variable "save_data : DLOG_SAVE" to be retained. At the end there are two cases:

- The file exist: Append the data
- The file doesn't exist: Create the file and write the data

I think this will be in general the better solution. Instead of doing so with retained variables, it would have the oportunity to store data to a file that wasn't created by the function DLOG_STORE_FILE_CSV itself. A switch to append or not to append in an alreadey existing file would be best of all.

My use case is to log process data and to have a look into it, if it's necessary. No chance to delete the file(s) during runtime. Therefore a style of shifting register, with an configurable absolute size/entry limit is needed. If for example 1000 lines are filled in the file, the oldest line has to be deleted and the new entry will be appended.

Can anybody at least help me changing the code, please?

PS: Gerne kann ich mich auch in Deutsch ausdrücken und dachte es würde wohl in Englisch eine Mehrzahl an Leute erreichen.

[peewit]

Therefore a kind of shift register is needed, with a configurable absolute size/input limit. If e.g. 1000 lines are filled in the file, the oldest line must be deleted and the new entry is appended.


this is not easily feasible and is a completely different concept
you can't just delete a few lines from a file
for this you have to load the complete file into a bytearray in the sps
then pick out the lines and move the rest of the data and save it again.

it would be easier if you use the "LOG_CONTROL" block right away
it has a ringbuffer - there happens what you want automatically

[laruso]

After a deep dive into the code I understand a few things and found a solution for my problem.

First thought was to modify the function block DLOG_STORE_FILE_CSV, but if the OSCAT library will be changed in the future, the function block will differ (I would like to prevent that scenario).
Second thought was to modify the call of the function block. To do so I did:

- Previous to the call of DLOG_STORE_FILE_CSV, a separate FILE_SERVER is used to try to open the file. When there is no error code, the file exist. Then only the DLOG_SAVE.FN_REM has to be manipulated by copying the filename to it. Then the function block DLOG_STORE_FILE_CSV is called and thinks that it was already written to this filename and therefore only append the additional entries. Thats like the filename was stored in the retained variable.
When there is an error while trying to open the file, the DLOG_SAVE.FN_REM will be erased and the enabled will be set to FALSE for some cycles/time. Therefore the function DLOG_STORE_FILE_CSV thinks, that there it wasn't written to that filename in the past. Therefore a new file will be created with new header line.

Sounds complicated and it is ;-) But it's (just) a preconfiguration for a non existent feature of DLOG_STORE_FILE_CSV. It would be nice if some can switch the functionality in the interface of the function call. To either use retained variables or directly check if the file exist at write request.

But I also would understand that this will complex the code to much by adding a feature select.

This is my code to preconfigure the implementation to add this feature.
Declaration:

Code Auswählen Erweitern

VAR
R_TRIG_LogExportRequest: R_TRIG;
xFileExistChecked: BOOL;
xFileExist: BOOL;
usiStep1: USINT := 1;
usiStep2: USINT := 1;
FS : OSCAT_NETWORK.FILE_SERVER;
FSD : OSCAT_NETWORK.FILE_SERVER_DATA;
PT : OSCAT_NETWORK.NETWORK_BUFFER;
trig_m : BOOL; // Manual trigger (for storage of values to new line of file)
trig_m_last : BOOL; // Last manual trigger (for storage of values to new line of file)
filename: STRING := 'Log'; // Name of file   'Log#A-#D-#H'
error_c: DWORD; // Corresponding error code
error_t: BYTE; // Corresponding error type
tonEnable: STANDARD.TON := (PT:=T#1000MS);
x1 : OSCAT_NETWORK.DLOG_DT := (COLUMN:='Time stamp'); // Logger input for STRING, used for Time stamp
x2 : OSCAT_NETWORK.DLOG_STRING := (COLUMN:='Event type'); // Logger input for STRING, used for Event type
x3 : OSCAT_NETWORK.DLOG_STRING := (COLUMN:='Description'); // Logger input for STRING, used for Description
x : OSCAT_NETWORK.DLOG_DATA; // Data exchange for logging and file storage
RTC_2: OSCAT_NETWORK.OSCAT_BASIC.RTC_2; // Clock needed for (tsv) file write
DLOG_STORE_FILE_TSV : DLOG_STORE_FILE_CSV; // File storage (tsv)
END_VAR

VAR RETAIN
save_data : OSCAT_NETWORK.DLOG_SAVE;
END_VAR


Code:

Code Auswählen Erweitern

R_TRIG_LogExportRequest(CLK:=(LogExport.uiTxBufferCount > 0));
IF R_TRIG_LogExportRequest.Q THEN
xFileExistChecked := FALSE;
xFileExist := FALSE;
usiStep1 := 1;
usiStep2 := 1;
END_IF


IF NOT xFileExistChecked THEN

CASE usiStep1 OF

01: FSD.MODE := 5; // Close the possibly opened file
FSD.FILENAME:='Log'; // with filename "Log"
usiStep1 := 2;

02: IF FSD.MODE = 0 THEN // Close operation finished
usiStep1 := 3;
END_IF

03: FSD.MODE := 1; // Open+read file
usiStep1 := 4;

04: IF FSD.MODE = 0 THEN // Open+read operation finished
usiStep1 := 5;
END_IF

05: IF FSD.ERROR = 0 THEN // Open+read w/o error = file exist
xFileExist := TRUE;
ELSIF FSD.ERROR = 5 THEN // Open+read w error = file not exist
xFileExist := FALSE;
END_IF
xFileExistChecked := TRUE;

END_CASE;

FS(FSD:=FSD,PT:=PT); // File server operation

END_IF



IF xFileExistChecked THEN

CASE usiStep2 OF

01: IF xFileExistChecked AND xFileExist THEN // File exist = enable (to only open+write)
enable := TRUE;
save_data.FN_REM := 'Log'; // copy the file name to remanent variable to prevent "create+write"
ELSIF xFileExistChecked AND NOT xFileExist THEN // File not exist = not enable -> enable (to create new file with header)
enable := FALSE;
save_data.FN_REM := ''; // delete file name in remanetn variable to allow "create+write"
END_IF
usiStep2 := 2;

02: IF NOT enable THEN // Reenable when preveiously disabled
enable := TRUE;
END_IF
usiStep2 := 3;

03: IF LogExport.uiTxBufferCount > 0 AND trig_m = trig_m_last AND tonEnable.Q THEN
trig_m := TRUE; // Set the trigger
END_IF
trig_m_last := trig_m;
tonEnable(IN:=enable);

END_CASE



x1(X:=x); // Logger input for STRING, used for Time stamp
x2(X:=x, STR:=LogExport.asTxContent[1].sLogExportText2); // Logger input for STRING, used for Event type
x3(X:=x, STR:=LogExport.asTxContent[1].sLogExportText3); // Logger input for STRING, used for Description
DLOG_STORE_FILE_TSV(X:=x, SAVE_DATA:=save_data, ENABLE:=enable, TRIG_M:=trig_m, FILENAME:=filename, DTI:=RTC_2.UDT, SEP:=9, AUTO_CLOSE:=TIME#10S, ERROR_C=>error_c, ERROR_T=>error_t); // File storage (tsv)

IF trig_m THEN
trig_m := FALSE; // Reset the trigger
FOR iI := 1 TO TO_INT(LogExport.uiTxBufferCount) - 1 DO // Shift every entry one index up
LogExport.asTxContent[iI].sLogExportText2 := LogExport.asTxContent[iI+1].sLogExportText2;
LogExport.asTxContent[iI].sLogExportText3 := LogExport.asTxContent[iI+1].sLogExportText3;;
END_FOR
LogExport.asTxContent[iI].sLogExportText2 := '';
LogExport.asTxContent[iI].sLogExportText3 := ''; // And fill the last one with an empty entry
LogExport.uiTxBufferCount := LogExport.uiTxBufferCount - 1; // And decrement the buffer counter
END_IF

END_IF