Slate AX CPU to 100 and all network traffic ceased reboot required

[thumper]

Not sure which logs to start digging in. /usr/local/share/speedigy/logs seems to have persisted past the reboot.

root@SmoothWAN:/usr/share/speedify/logs# ls -ltra
drwxrwxr-x    3 root     root           976 Apr 12 20:05 ..
-rw-r--r--    1 root     root        173563 Apr 13 23:59 summary_041323.txt
-rw-r--r--    1 root     root          1677 Apr 14 21:19 histogram_20230413_145654.tsv
-rw-r--r--    1 root     root        265356 Apr 14 23:52 summary_041423.txt
-rw-r--r--    1 root     root          2748 Apr 15 20:03 histogram_20230414_211943.tsv
-rw-r--r--    1 root     root       1060784 Apr 15 20:09 speedify_20230415_200353.log_0
-rw-r--r--    1 root     root        419993 Apr 15 20:09 speedify_20230415_200910.log_0
-rw-r--r--    1 root     root       3145728 Apr 15 20:46 speedify_20230415_200950.log_2
-rw-r--r--    1 root     root       3145728 Apr 15 21:10 speedify_20230415_200950.log_1
-rw-r--r--    1 root     root       1904788 Apr 15 21:23 speedify_20230415_200950.log_0
-rw-r--r--    1 root     root          2628 Apr 15 21:24 histogram_20230415_200950.tsv
-rw-r--r--    1 root     root        604847 Apr 15 23:59 summary_041523.txt
-rw-r--r--    1 root     root        332308 Apr 16 23:50 summary_041623.txt
-rw-r--r--    1 root     root        346793 Apr 17 23:50 summary_041723.txt
-rw-r--r--    1 root     root       3145728 Apr 18 15:46 speedify_20230415_212412.log_2
-rw-r--r--    1 root     root       3145728 Apr 18 15:55 speedify_20230415_212412.log_1
-rw-r--r--    1 root     root        846943 Apr 18 15:57 speedify_20230415_212412.log_0
-rw-r--r--    1 root     root        108945 Apr 18 15:58 settings.txt.lastGoodRead
-rw-r--r--    1 root     root            46 Apr 18 15:58 VERSION
-rw-r--r--    1 root     root          2476 Apr 18 15:58 histogram_20230415_212412.tsv
-rw-r--r--    1 root     root       3145728 Apr 18 16:31 speedify_20230418_155842.log_2
-rw-r--r--    1 root     root        359007 Apr 18 16:32 summary_041823.txt
-rw-r--r--    1 root     root       3145728 Apr 18 16:36 speedify_20230418_155842.log_1
-rw-r--r--    1 root     root        109201 Apr 18 16:39 settings.txt.lastGoodWrite
-rw-r--r--    1 root     root        109201 Apr 18 16:39 settings.txt
-rw-r--r--    1 root     root         27256 Apr 18 16:39 lastUsedSettings.txt
-rw-r--r--    1 root     root        151552 Apr 18 16:39 sql.db
drwxr-xr-x    2 root     root          2416 Apr 18 16:39 .
-rw-r--r--    1 root     root       1415326 Apr 18 16:39 speedify_20230418_155842.log_0
root@SmoothWAN:/usr/share/speedify/logs# 

Side note: It would be good to be able to specify a log directory e.g. mounted sd-card.

Symptom was that web browser wouldn’t load anything from the internet. I could still (slowly) access luci and netdata.

Attached some graphs from netdata that I was able to grab before reboot. Didn’t have a console to check which proc(s) were on top.

[TalalM]

That’s interesting, can you share what other features enabled other than Speedify, and an estimate of connected Wi-Fi clients?
I think you can mount /var to a pre-formatted ext4 SD-card in LuCI mountpoints menu for logs.

[thumper]

I’ve only added netdata, luci-app-nlbwmon and updated some filesystem utils
There are three wifi clients on Narwhal
Zero on shadow
Maybe ten on raven

root@SmoothWAN:~# cat /etc/device_info 
DEVICE_MANUFACTURER='OpenWrt'
DEVICE_MANUFACTURER_URL='https://openwrt.org/'
DEVICE_PRODUCT='Generic'
DEVICE_REVISION='v0'
root@SmoothWAN:~# cat /etc/openwrt_release 
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='21.02-SNAPSHOT'
DISTRIB_REVISION='r16399+159-c67509efd7'
DISTRIB_TARGET='ipq807x/ipq60xx'
DISTRIB_ARCH='arm_cortex-a7'
DISTRIB_DESCRIPTION='OpenWrt 21.02-SNAPSHOT r16399+159-c67509efd7'
DISTRIB_TAINTS='no-all override'
DISTRIB_TIP='OpenWrt 21.02-SNAPSHOT r16399+159-c67509efd7 / TIP-devel-cb0d6246'
DISTRIB_TIP_VERSION='devel'
DISTRIB_DESCRIPTION='SmoothWAN v0.99.5HF5'
root@SmoothWAN:~# cat /etc/openwrt_version
ApNos-cb0d6246
r16399+159-c67509efd7
root@SmoothWAN:~# /usr/share/speedify/speedify_cli version
{
	"bug" : 1,
	"build" : 10893,
	"maj" : 13,
	"min" : 2
}
root@SmoothWAN:~# 

[thumper]

of course now that I’m watching it constantly there’s not been any problems.
Added vnStat2
Also added the bandwidth sharing under SQM as described on the wiki

[thumper]

I did have a low memory banner pop up in the speedify UI about an hour ago right around Uptime: 2 days. When I checked, speedify was top of memory use. And one CPU was showing pegged with speedify -d logs as the top process both CPU and Memory. I did /etc/init.d/speedifyconf restart and the memory use went down and the speedify -d logs is now in check

[TalalM]

I haven’t updated to the latest version, feel free to downgrade Speedify version in the Options menu, I hope it’s not an issue with the latest update.

Similar issue: Speedify Stops and cannot be restarted · Issue #141 · SmoothWAN/SmoothWAN (github.com)

[TalalM]

Looks like zram has to be enabled for the latest release, unless other processes have abnormal memory usage.

Use htop command & press F6 to sort by memory usage to see if other processes have abnormal memory usage.

Related:
Speedify memory leak/usage issue · Issue #89 · SmoothWAN/SmoothWAN (github.com)

[TalalM]

Enabling zram next release, upon further tests, Flint & Slate AX 5Ghz Wi-Fi driver use too much memory that is not freed when 10+ clients are connected.

Flint/Slate AX patch for testing, in terminal, type the following to enable zram:
Slate:

wget https://raw.githubusercontent.com/SmoothWAN/SmoothWAN/main/devconfigs/slateax/files/etc/rc.local -O /etc/rc.local

Flint:

wget https://raw.githubusercontent.com/SmoothWAN/SmoothWAN/main/devconfigs/flint/files/etc/rc.local -O /etc/rc.local

[thumper]

It’s just speedify process that starts consuming ram. At about the two day mark I start getting low memory banner in the UI.

[TalalM]

Hope zram helps, else you may need to replicate the memory (leak?) issue on Ubuntu then send the logs to Speedify, since they do not support OpenWrt.

[thumper]

With zram enabled and swap on, used RAM seems to be holding steady now at the ~71% mark and free does show some swap in use. Before zram, all swap numbers were zeros.

root@SmoothWAN:/usr/share/speedify# free
              total        used        free      shared  buff/cache   available
Mem:         402612      278336       33768         448       90508       76044
Swap:        195312        5636      189676
root@SmoothWAN:/usr/share/speedify# 

[thumper]

In terms of zram, it looks like there are luci settings instead of forcing something in rc.local wouldn’t it make more sense to use uci to keep it more obvious?

[TalalM]

Good to hear, not sure if that option works on the unofficial OpenWrt build by GL.iNet, let me know if it does. Only one of the compression algorithms is available.
Will push a hotfix soon with zram enabled.

[thumper]

@TalalM

Only one of the compression algorithms is available.

[TalalM]

As a kernel module I mean. That’s LuCI support for the official builds. (gl.inet is not official, and smoothwan is using that)

[thumper]

Memory usage has been been climbing.

root@SmoothWAN:/usr/share/speedify/logs# uptime
 22:50:07 up 4 days, 41 min,  load average: 0.49, 0.84, 0.96

Speedify was 196 or so yesterday

And after /etc/init.d/speedifyconf restart we have this

and whatever was swapped out is probably not speedify related

root@SmoothWAN:/usr/share/speedify/logs# free
              total        used        free      shared  buff/cache   available
Mem:         402612      217064      105380         528       80168      137320
Swap:        195312        5404      189908

[TalalM]

I wonder if the previous smoothwan versions are different, since I noticed that the official firmware 4.2.0 is using a bit more memory than 4.1 , and HF5 use the 4.2 base.
Don’t remember what Speedify usually used in terms of RAM unfortunately.
I do remember that it never reached hundreds though.