Milvus Metrics - Could Not Open Hwmon Directory · Discussion #15885

Trang chủ » C5 9xl » Milvus Metrics - Could Not Open Hwmon Directory · Discussion #15885

Có thể bạn quan tâm

Skip to content Dismiss alert {{ message }} / milvus Public
  • Notifications You must be signed in to change notification settings
  • Fork 3.7k
  • Star 42k
  • Code
  • Issues 741
  • Pull requests 155
  • Discussions
  • Actions
  • Projects
  • Wiki
  • Security

    Uh oh!

    There was an error while loading. Please reload this page.

  • Insights
Additional navigation options Milvus Metrics - Could not open hwmon directory #15885 Unanswered durai89 asked this question in Q&A and General discussion Milvus Metrics - Could not open hwmon directory #15885 @durai89 durai89 Mar 6, 2022 · 4 comments Return to top Discussion options {{title}} Something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Quote reply edited {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

durai89 Mar 6, 2022

After we enabled metrics in server_config.yaml, we are seeing loads of error messages like ERROR][SERVER][CPUTemperature][metric_thread] Could not open hwmon directory .

System Spec:

  • Amazon EC2 c5.9xl instance, centos7
  • Milvus container: milvusdb/milvus:1.1.1-cpu-d061621-330cc6

Can you please advise what's the best approach to fix this issue? lm_sensors module is installed but is unable to detect any sensors on this ec2 instance. Though it would be great to report this metric, we are also happy if there is a way to blacklist this specific metric alone, so that our logs don't get bombarded with this error message?
1 You must be logged in to vote All reactions

Replies: 4 comments

  • Oldest
  • Newest
  • Top
Comment options {{title}} Something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Quote reply edited {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

LoveEachDay Mar 7, 2022 Maintainer

Please make sure the /sys/class/hwmon directory exists, according to #4240

If you have installed lm_sensors, you can run sensors command to check whether any sensors enabled.
1 You must be logged in to vote All reactions 0 replies Comment options {{title}} Something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Quote reply

yhmo Mar 7, 2022 Collaborator

The source code use a hard code path to get CPU temperature from "/sys/class/hwmon/"

std::string path = "/sys/class/hwmon/"; DIR* dir = nullptr; dir = opendir(path.c_str()); if (!dir) { LOG_SERVER_ERROR_ << "Could not open hwmon directory"; return result; }

Maybe the EC2 instance has no such directory. Is it possible to create an empty directory?
1 You must be logged in to vote All reactions 0 replies Comment options {{title}} Something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Quote reply

yhmo Mar 7, 2022 Collaborator

If the EC2 instance doesn't allow creating a directory under the /sys/class, the error cannot be avoided. You have to modify the source code and compile/build a new docker image for yourself. The steps:

  1. clone milvus project into your local git clone [email protected]:milvus-io/milvus.git

  2. switch to v1.1 branch and checkout to the v1.1.1 commit id git checkout 330cc61bede475c4a7a71841d54e633586cea829

  3. remove the error log in this cpp file: /milvus/core/src/metrics/SystemInfo.cpp

if (!dir) { // LOG_SERVER_ERROR_ << "Could not open hwmon directory"; return result; }
  1. compile milvus under the core folder
cd core ./build.sh -t Release
  1. build a new docker image
cd /milvus/docker/deploy/cpu/ubuntu18.04 docker build -t milvusdb/your-milvus:1.1.1-cpu .
  1. deploy the image to the EC2
1 You must be logged in to vote 👍 1 All reactions
  • 👍 1
0 replies Comment options {{title}} Something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Quote reply edited {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

durai89 Mar 7, 2022 Author

Thanks @LoveEachDay and @yhmo . Yes, I had already tried to create an empty directory /sys/class/hwmon but ec2 doesn't allow it which makes sense actually. I will try today to build the image commenting out the block of code 👍
1 You must be logged in to vote All reactions 0 replies Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment Category ❓ Q&A and General discussion Labels None yet 3 participants @durai89 @LoveEachDay @yhmo Heading Bold Italic Quote Code Link Numbered list Unordered list Task list Attach files Mention Reference Menu Select a reply Loading

Uh oh!

There was an error while loading. Please reload this page.

Create a new saved reply 👍 1 reacted with thumbs up emoji 👎 1 reacted with thumbs down emoji 😄 1 reacted with laugh emoji 🎉 1 reacted with hooray emoji 😕 1 reacted with confused emoji ❤️ 1 reacted with heart emoji 🚀 1 reacted with rocket emoji 👀 1 reacted with eyes emoji You can’t perform that action at this time.

Từ khóa » C5 9xl