Problems with upgrades to 0.3.x (errors during upgrade and no audio)

Folks,

I had not used the Amplipi system for quite some time and when I finally went to use it on 20 January and I noticed it had upgraded to 0.3.0 and I saw that the 0.3.1 upgrade was available so I installed the update BEFORE trying to play any music. I then tried to play pandora streams and, while it gave every indication of playing, it produced no audio. I then tried various downgrade/upgrade cycles to no avail as I would get errors programming the preamp boards like the below during upgrade and still had no audio.

I noticed the thread “Issues after upgrading to 0.3.0” started by jimi (Jim C) on 23 December and decided to reimage with 0.3.1 image since my unit is a pretty early one and I DID see differences in config.txt that were mentioned in the responses from linknum23 (Lincoln).

After reimaging I notice that the Amplipi system will play for some time and then just stop playing. I tried downgrading to 0.3.0 and downgrade failed on the preamp upgrade as noted below. The updater shows “Failed, Retry?” button as it has during yesterday’s pre-reimagine exercise so I hit it which is apparently going to install 0.3.1. After a reboot it seemed to work fine DESPITE the errors during the upgrade but today it does not want to play audio.

It all worked just fine yesterday although it stopped producing audio a few times despite giving every indication that it was playing and I had to change the stream to get it to play.

As mentioned, today it does not want to play audio and I saw that 0.3.2 has been released so I installed it and saw the same type of upgrade failure and same behavior with appearing to play but not actually producing audio.

Snippet of upgraded error messages included below. Screenshot attached as well. Log files attached as well.

After I post this message I will wait a bit to see if you all want me to gather any other data before do a full reboot.

Thanks,
Tim

10:14:54: installation failed
10:14:54:
Programming /home/pi/amplipi-dev/fw/bin/preamp_1.7.bin with
/home/pi//venv/bin/python -m amplipi.hw --baud 115200 --flash "/home/pi/amplipi-dev/fw/bin/preamp_1.7.bin"

Failed to init device.
stm32flash 0.5

http://stm32flash.sourceforge.net/

Using Parser : Raw BINARY
Interface serial_posix: 115200 8E1

Programming up to 6 AmpliPi Preamps (3 currently detected)

Main Unit's old firmware version: 1.7-4F618C6
Error programming Main Unit, stopping programming
Main Unit's new firmware version: 1.7-4F618C6

0 AmpliPi preamps programmed.
Programming failed.




Hey Tim,

I’m sorry you’re experiencing issues after the update. I unfortunately likely can’t dig into this until later this week but here’s my initial thoughts. The firmware update failing is definitely annoying since it’s causing the update to throw a failed status, but all the firmware versions are compatible with each other and this update was just to handle some hardware changes on Preamp boards shipping starting 2 months ago. Also from your log, at least the Main Unit has been updated to the latest firmware, 1.7-4F618C6. Based on all this info, I think the firmware update failures are a not the root cause of your lack of audio.

What would be really helpful to further diagnose this is a full log when the issue occurs (settings->about->logs), and your config file (settings->configuration->download config). Also, the most useful tidbit of what you mentioned was:

I had to change the stream to get it to play.

which stream are you changing, and to what? Are you adding the new stream, then removing the old one? Or are you removing the failed one, then adding a new one?

Thanks,
Michael

Michael,

Thanks for the reply.

A few things.

I have various log files saved that I can share if you tell me how to get them to you.

The only thing that jumps out at me in the logs is this recurring message:

Jan 30 10:11:44 amplipi authbind[841]: alsaloop: pcm.c:1654: snd_pcm_poll_descriptors_count: Assertion `pcm' failed.
Jan 30 10:11:44 amplipi authbind[841]: Subprocess exited with non-zero code: -6. Restarting...

I have no idea if that is normal behavior but it occurs periodically as show in all the log files I have looked at.

Additionally, when I said “I had to change the stream” I did not actually edit the settings for the stream. I just selected a different stream to play.

It seems to be working normally now for several days most of the time after a cold power cycle of all components. Today it was playing a pandora stream and just stopped producing any audio and I saved a log file and looked at it and nothing jumped out at me.

I then saw that another upgrade (0.3) was out so I went a head and upgraded to it and it would not produce audio after the upgrade so I rebooted the Pi and now it seems to be working fine.

Today’s upgrade successfully upgraded the fw on 2 of 3 preamps (I have 2 zone expanders). Someone should probably sort out what is going on with that at some point.

So I can send you log files now or I can wait until the system stops playing a stream again and send log files then whichever you prefer.

PLMK

Thanks,
Tim

Hey, sorry for the delay getting back to you Tim, I was out of town for the last few days.

I’m glad to hear things seem to be working a bit better now with the latest update. We did fix an issue recently where for some AmpliPis the built-in Raspberry Pi audio was still being loaded and it broke our audio setup, and your log looks similar to issues we’ve seen from that. Also, since the firmware failed to program I believe the updater will exit early instead of doing a reboot to finish the update. So that would explain why you needed to reboot to get audio. I’ll have to have someone look into that if my hunch is correct.

If the issue occurs again in the future, you can send us the description of the problem (or I guess just link to this discussion if it’s the same problems) and the logs at support@micro-nova.com.

As far as the firmware programming goes, unfortunately the failures with multiple expanders are a known issue and I’m waiting on enough hardware being built to fully test, see Robustify firmware programming when expansion units present · Issue #587 · micro-nova/AmpliPi · GitHub for updates. As I said before though, you shouldn’t suffer any problems with the older firmware on those expanders.

Just to follow up here, the root cause of the issue has been fixed and will be included in the next release. Unfortunately there may still be some difficulties getting 1.8 programmed in the first place. We’ve robustified the programming script to go from a ~10% success rate in your case of 2 expanders to ~90% success rate, but that means there is still a chance of failure. If you do see errors related to the preamp firmware programming during future updates it’s not a big deal, the only change to 1.8 is the programming fix and in general we make the preamp firmware releases backwards-compatible.

Is there a workaround?

Like say maybe disconnect one of the expanders and update and then switch and do the other?

According to the last log you shared your preamp firmware is already up to date, so there’s nothing to work around. It’s unfortunately just an annoying bug that will make updates appear to have failed even though they technically are OK.

If you’re still getting audio playback issues let us know, but that would be an unrelated issue to the preamp firmware upgrade errors.

That is good to know. Thanks. Also Thanks for the follow-up on general.

Still have audio just drop out a fair bit.

We only really play Pandora and I saw there were some updates to make Pandora streams more reliable recently. I guess I need to set up some other streams and see if they drop out too.

Howdy @tgsmith -

I just stamped a pandora-hotfix alpha release and was wondering if you could give this a go and let us know if it works better for you? You can find this in the Settings → Updates → Older Releases tab, and it should be at the very top of the list.

We had occasionally observed instability with Pandora after making those changes you mentioned, but it was pretty hard to pin down and we weren’t sure if we were simply re-using our internal Pandora account too much :sweat_smile: We’d love at least one in-field report that reverting those “stability” changes corrects this. As a side benefit, this prerelease has the new firmware flashing methodology baked-in, which seems like something you may appreciate.

Thanks for your communication and for working with us!

Hey Tim,
I noticed this at home on my system as well. Turns out I was using the same Pandora account in multiple places. Once I stopped using the same account to stream at the same time, the problem went away - just an FYI, not sure if that’s your issue.
Jason

I am currently on the road but will be home late Thursday and will try this Friday.

Thanks.for digging into this and I am very happy to help in anyway I can.

I do not believe it to be this same issue since we really only use the Pandora account for AmpliPi

Do you use multiple Pandora streams simultaneously with the same account?

Installed the update.

Amps updated properly. Yay. I hate messy upgrades even if “they don’t matter”. I did firmware back in the 90s and burnt days and days tracking down timing and other BS issues because I just could not help myself. So thanks!!!

Testing Pandora now/this week. Will let you know how it goes.

Thanks for looking into it and making progress.

Fighting entropy is futile yet necessary…

howdy @tgsmith !

how has that patched version been working for you?

Much better. Thank you!

Following up was on my list for today/tomorrow but you beat me to it.

Thanks so much for your feedback here! We released that bugfix with version 0.3.5 just now, along with a handful of other fixed and niceties.