GoogleCloudPlatform/market-data-transcoder

Print interim statistics to STDERR on demand

Open

#111 opened on Mar 5, 2023

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (8 forks)auto 404
enhancementgood first issue

Repository metrics

Stars
 (47 stars)
PR merge metrics
 (PR metrics pending)

Description

One could run the transcoder with debug options, but this slows down processing since the output is a function of a file. In some instances, one might desire seeing where a txcode process is with it's current batch non-destructively. One way to do this might be trapping kill -s INFO signal in main thread and then dump a snapshot of the progress up to that point. The dd command has a similar feature:

Sending an `INFO' signal to a running `dd' process makes it print
I/O statistics to standard error and then resume copying.  In the
example below, `dd' is run in the background to copy 10 million blocks.
The `kill' command makes it output intermediate I/O statistics, and
when `dd' completes, it outputs the final statistics.

     $ dd if=/dev/zero of=/dev/null count=10MB & pid=$!
     $ kill -s INFO $pid; wait $pid
     3385223+0 records in
     3385223+0 records out
     1733234176 bytes (1.7 GB) copied, 6.42173 seconds, 270 MB/s
     10000000+0 records in
     10000000+0 records out
     5120000000 bytes (5.1 GB) copied, 18.913 seconds, 271 MB/s

Contributor guide