Running reliably using wrapper etc

[benpollinger]

Hi @simon3270 thanks for this script.

I have the UDP script running OK, but wondered if you had any pointers on keeping it running.

I'm on a Rasp Pi, mainly running OpenMediaVault. I made a bash script as you suggested:

while true; do
python3 ginserv.py >> gindata.log_$(date '+%Y%m%d')
sleep 30
done

I launched screen and ran the script in that. This seemed OK, but today I checked the log file and it didn't start til 5pm. I'm not sure what makes it stop/start/wobble. The data on the Ginglong site is continuous and the Rasp Pi was running OK today.

Anyway, any suggestions welcome.

[simon3270]

Hi Ben,

Thanks for using my script!

It's odd that it waited so long before starting. Was there more than one log file?

Mine start with something like

Socket created
Socket bind complete
Sat Oct 9 08:13:05 2021 1633763585
a5:01:00:10:47:0d:53:8a:1e:ca:71:00:9b:15

There should be one created as soon as you run the loop (I just run it in the background, without screen). The ginserv.py then runs until it reads 90 records, the exits. The loop starts it again, writing a new log file, 30 seconds later (to give the network time to settle down).

I think the connection is a bit fragile. I have a script which looks for the most recent log file - if it is not bigger than the initial 82 bytes that the script writes, it kills the current ginserv.py (then the loop starts another). I run this every hour during the morning, in case logging doesn't start. I'll makes sure that's all in git over the next few days.

At least you are doing this in the daytime. I started working on the script in winter evenings, and it took me an embarrassingly long time to realise that I wasn't getting messages because it was dark :-)

[benpollinger]

Hi, thanks for the reply,

No, only one file yesterday, I was at work so not caused by my tinkering! I left it running overnight, it started up as expected this morning with the 'Socket...' lines. I'm using screen as I'm using putty to SSH to my rasp pi, I wasn't sure if the wrapper would keep going when I turn off my laptop.

It would be handy for me to have a wrapper which starts a new dated log each day (hence I was trying python3 ginserv.py > gindata.log_$(date '+%Y%m%d') as above).

I know what you mean with the winter problem - I've been racing to try things out before the inverter shuts down each evening!

[simon3270]

Yes, the loop will continue (I start mine with a script in crontab with "@reboot"). I also have a crontab entry to kill the ginserv.py just before midnight, so that a single run covers just one day.

There's definitely more I need to add to my github. I'm away from home this weekend, but I will get onto it on Monday.

[simon3270]

Ah, just noticed. If you only have YMD for the filename, use >> to write to the file to append to an existing one, otherwise each run will overwrite the previous one that day. That's why I include the time in the file name. I usually have at least 2 files per day, sometimes several!

[benpollinger]

That crontab sounds handy for me too.

I realised that another option is have one big log file, process it all with get_data.py, then parse out each new day's data to a new date-named file using a bash script. (I have no python knowledge but am learning to use bash)

I did use >> to append, I think I just copy-pasted your original for the above post and missed it!

[simon3270]

As promised, support bash scripts, and crontab entries, added in the "support" directory!

[benpollinger]

Great, thank you! I will try these out.

[benpollinger]

These seem to work well @simon3270.

I have to run the midnightcheck as root in crontab, as it calls the kill script with sudo which fails on my system otherwise (I'm using OpenMediaVault on Raspbian buster). I don't suppose it's a problem.

[simon3270]

Glad they were useful! You can actually remove the sudo - I think that was from early experiments, where I was sometimes running the polling program as root. Now that it's running as 'pi', the sudo isn't required.

[benpollinger]

Thanks again @simon3270 I've been using these without a problem all week.

I've built a little on your script in my fork, to send daily data to pvoutput.org - https://github.com/benpollinger/ginlong2pvouput

I don't know python really but have grasp enough of Bash commands to get this doing what I needed, so far so good.