Repository navigation
[FEATURE][ChromeOS] Expose DSP load tracker info for the visibility to ChromeOS #5521
Description
Is your feature request related to a problem? Please describe.
Currently we don't have much information for DSP in system logs attached in feedback report of audio issue. To debug the issue we need to re-build kernel with trace feature on and capture logs by sof-logger. It will be super helpful for debugging if DSP load tracker info could be obtained without those debugging steps. Moreover, we can setup an autotest item to track the DSP load performance for all supported products.
Describe the solution you'd like
I don't have ideas which should be the appropriate/simple way to do this. It can be either actively reported from DSP, or on demands of the host request.
For the active approach, it should be enough by reporting critical info only (e.g. DSP load is over the threshold) in order not to cause more burdens.
Describe alternatives you've considered
None
Additional context
None
Activity
lgirdwood commentedon Mar 11, 2022
@plbossart @ujfalusi @ranj063 any thoughts ?
kv2019i commentedon Mar 25, 2022
When this is implemented, feature needs to be documented to https://thesofproject.github.io/latest/developer_guides/debugability/perf-counters/index.html?highlight=performance
kv2019i commentedon Mar 25, 2022
I'd say an IPC to on-demand ask this information from FW would be the most suitable solution. So we could e.g. have a debugfs interface (exposed by SOF linux driver) and when CRAS queries this (at the rate it chooses), SOF drivers sends an IPC and FW replies with current DSP load tracker data.
lgirdwood commentedon May 11, 2023
@kv2019i we have mtrace and Zephyr oops now, anything else pending ?
btian1 commentedon May 12, 2023
there is one line log may cover this topic:
[ 65.782163] ll_schedule: ll timer avg 2138, max 4647, overruns 0
[ 66.806163] ll_schedule: ll timer avg 2143, max 2924, overruns 0
simply check overruns can know whether dsp overloaded.
kv2019i commentedon May 12, 2023
@lgirdwood @btian1 This is not done. The request is to have this data in user-space without enabling the FW logs (disabled in Chromebooks by default). So this requires new DSP+kernel infra. In IPC4, we have messages to report load measurements, but these are not implemented yet (on FW nor on kernel side). So this is still a valid ask and I'm not aware of any ongoing work to implement these (we discussed about the IPC4 messages in question with the cycles-budget (CPC) topic around manifest).
marc-hb commentedon May 12, 2023
Why is this not just a log level configuration problem?
https://docs.zephyrproject.org/latest/services/logging/index.html
The old sof-logger has similar capabilities.
So the idea to start another, new and entirely different logging framework sounds weird to me. @johnylin76, @andyross, @cujomalainey why are Chromebooks "throwing the baby with the bathwater" and disabling logs entirely as opposed to selectively keep the logging they need/want?
cujomalainey commentedon May 15, 2023
@marc-hb at least on pre-ace systems I have been warned that enabling logging hijacks a DMA (which PNP is not happy about when it comes to power) and also it can block power states. Trust me I would love to have even just ERR/WARN on.
marc-hb commentedon May 15, 2023
The old sof-logger uses either shared memory for high levels or DMA for all levels or both at the same time. It's configurable.
I'm not familiar with all Zephyr logging possibilities yet but I know for sure that the way it's configured does NOT use any DMA right now in our CI, it relies on shared memory "mtrace" only (which means it can overrun and lose statements; "you get what you paid for")
kv2019i commentedon May 17, 2023
@cujomalainey @marc-hb That's a good question. "mtrace" Zephyr backend is friendlier to system flows (it doesn't take a DMA channel). but it's still not zero cost. There are writes to the SRAM that take DSP cycles, there's a logging thread that takes some memory and there's impact to code size (although we already have dictionary formats in Zephyr up and running limiting this). But in the end it will depend on the specific product. Our current assumption is still that at least not all release firmwares will have logging enabled (like has been the case with SOF previously). I.e. key debugging tools (like ability to extract a FW coredump), will be available even if you don't have logging enabled. Having a way to access DSP load information without logging, seems like a reasonable feature, similar to the coredump. I think we are running out of runway for v2.6 though, so this probably needs to be pushed out to v2.7.
32 remaining items
marc-hb commentedon Apr 30, 2024
Again, not sure about this particular case but a typical configuration for this type of features in general is:
Problem solved?
plbossart commentedon Apr 30, 2024
how would we know at the kernel level if the feature is on or off by default? There's a need to report this capability...
johnylin76 commentedon Apr 30, 2024
I second @marc-hb
I'm so glad to see this feature being fulfilled. Since it's a developer feature anyway, Kconfig makes more senses to me as well.
My two cents is that kernel doesn't really to know whether this feature is supported or not, as long as no false alarm is reported by firmware while it does not support.
In another aspect, kernel can never consider the feature is "on" by default because there is always the mismatch between two ends.
plbossart commentedon Apr 30, 2024
in my experience, collecting data after a random time isn't great for early boot issues.
If you don't start from T0, then your results will depend on when the host driver sends that IPC, and that depends on a user-space program which is even worse.
Also there's a conceptual difference between COLLECTING data and REPORTING data. The first one should not be dependent on anything.
lgirdwood commentedon May 1, 2024
All, lets take this step by step - we can incrementally add to the config/control when the basics are upstream.
Today, we just need the Kconfig that advertises (enabled) in the extended manifest.
Next steps can be IPCs for runtime config/control, different backends (like probes) and so on, the important part is we need the basics 1st.
marc-hb commentedon May 1, 2024
I think everyone is on board with incremental, Continuous Integration but to reduce churn, back-and-forth and even going around in circles sometimes, it's good to have some high level idea of the final design and UI a.k.a. "requirements"? They don't have to be set in stone of course...
"Fail to plan, plan to fail" - my 2 cents.
lgirdwood commentedon May 1, 2024
See above re UI #5521 (comment) requirement is to track DSP load as per subject.
marc-hb commentedon May 1, 2024
I was including the user interface in the "requirements" but I admit none of this is an "exact science".
kv2019i commentedon Jun 5, 2024
There are two solutions now to expose DSP load over share memory:
kv2019i commentedon Sep 12, 2024
I think we are ready now v2.11 work with #9398 and #9451 . @jsarha can you add a summary of what is available now and how to use the feature?
@johnylin76 if you can review and see if this meets the original feature request needs, or do we something left to do under this ticket (we can then either close, move to 2.12 and/or file follow-up new tickets).
johnylin76 commentedon Sep 12, 2024
LGTM. Thanks for the great work.