Hi All,
Was able to solve this issue by updating to 396 from 390. The following resources have been very helpful for resolving this. Hoping this may help others in the future.
- https://docs-dev.fast.ai/troubleshoot#correctly-configured-nvidia-drivers - Overall steps to be followed but not complete as we need to figure out the specific driver installation part based on the platform being used(OS)
- https://medium.com/@philliplakis/removing-nvidia-drivers-on-centos-7-for-upgrading-bf00a2a5f0df - Mostly followed the instructions in this blog for the installation. (There is a catch though - the uninstallation of old drivers(390) was not working as it couldn’t recognise thru the uninstall command though nvidia-smi was working with 390 by then). we went ahead and tried installing 396 anyways and it worked. And answered yes to all the prompts in between (the dkms kernel builds etc)
- https://www.kinetica.com/docs/install/nvidia_rhel.html - Can fallback to this if above blog is not helpful
Thanks @shriram.jaju for the help