Conda wouldn't install sundials and scikits.odes

I wanted to install a new integrator for my dear python, since scipi odeint was running crappy on my equations.  I tried to install scikits, but I kept on running into errors:

"ImportError: No module named 'scikits'"

Then after trying to install it, I got:

"Conda package install [Errno 13] Permission denied"

and then after upgrading pip:

"fatal error: sundials/sundials_types.h: No such file or directory"

First, I tried to manually install sundials from https://github.com/certik/sundials which I did with configure, make and make install, but the error in conda persisted. Then I did "export SUNDIALS_INST=/usr/local/include/sundials/" but conda didn't like it. After some digging, I've found it's better to install sundials with pip. So I first had to remove my manual installation:

sudo rm -R /usr/local/include/sundials  

unset SUNDIALS_INST=/usr/local/include/sundials/
 

Then I decided to clean up conda a bit:

conda clean -a -y 

Then I upgraded pip:

pip3 install --user --upgrade pip #that didn't work so:

sudo pacman -S pip
conda install pip 

Then I installed sundials:

pip3 install --user Sundials

sudo pacman -S sundials #that worked
 

Then installed just in case scikit-learn:

sudo pacman -S python-scikit-learn
 

To install it from pacman:

sudo pacman -S sundials

conda -c install conda-forge sundials

conda install -c conda-forge cython  

conda install -c conda-forge scikits.odes #see here for the explanation

sudo conda install -c conda-forge scikits.odes #this took super long because it verified dependencies

pip3 install --user scikits.odes #this one did it
And now, it's installed and loadable.
(see also https://github.com/bmcage/odes/issues/43)

Nvidia on Manjaro

Short note on Nvidia drivers, following this guide:

Apparently in Manjaro, you have to install it with

$ sudo mhwd -a pci nonfree 0300

But this won't give you nvidia-smi. To get it, you need to also install the driver from Pacman: 

sudo pacman -S lib32-nvidia-utils
sudo pacman -S nvidia-utils
sudo pacman -S linux60-nvidia

Then if you're lucky, it may work and you can switch to it with:

$optimus-manager --switch nvidia

If you're feeling naughty and you want to use another kernel, like xanmod, you do the same, but with nvidia-dkms and you have to install also linux-headers before.

Useful commands (source):

lshw -C display

glxinfo -B

xrandr --listproviders  

How to remove old kernel entry in GRUB (non-funcitonal one)

   This is what I did to remove an old, non-functional kernel from my GRUB. I don't know how it even got there on the first place, because I had another, working xanmod entry. 

In any case, please proceed with caution when doing it. 

    ls /boot/vml*
  sudo mv /boot/vmlinuz-manjaro-xanmod ~
  ls /boot/vml*
  sudo grub-mkconfig -o /boot/grub/grub.cfg

The cursed "Intel Wi-Fi 6 AX200 "

So this are my notes on the cursed wifi adapter "Intel Wi-Fi 6 AX200 ".  It has the interesting tendency  to disappear and then you have to reboot for it to reappear. But sometimes it will again disappear just minutes after restart. So a really really naughty adapter. I spent a lot of time searching for a solution, here is what I found.

 The errors:

Error sending STATISTICS_CMD: time out after 2000ms.
[23658.567163] iwlwifi 0000:03:00.0: Current CMD queue read_ptr 56 write_ptr 57

or 

Direct firmware load for iwlwifi ucode failed with error -2

or 

iwlwifi 0000:03:00.0: Error, can not clear persistence bit

There are two parts of the problem that I detected so far.

1) Power management for optimus-manager. That's a tricky one because it appears if it's not configured correctly, at some point the wifi adapter shuts down. 

So this is my config file:

$cat /etc/optimus-manager/optimus-manager.conf
[amd]
dynamic_power_management=fine
DRI=3
driver=modesetting
tearfree=

[intel]
dynamic_power_management=fine
DRI=3
accel=
driver=modesetting
modeset=yes
tearfree=

[nvidia]
dynamic_power_management=fine
DPI=96
PAT=yes
allow_external_gpus=no
ignore_abi=no
modeset=yes
options=overclocking

[optimus]
dynamic_power_management=fine
auto_logout=yes
pci_power_control=no
pci_remove=no
pci_reset=no
startup_auto_battery_mode=integrated
startup_auto_extpower_mode=hybrid #nvidia
startup_mode=hybrid #nvidia
switching=none

2) The more important problem appears to be the wifi firmware. For unknown to humanity reasons, the kernel doesn't install the correct firmware but 20 others. This page expains how to reinstall it.

So  first see the exact adapter you have, unless you haven't.  In Manjaro:

$inxi -N

or 

$lshw -c net 

Then you can check the firmware version with:

$modinfo iwlwifi|grep 7260

Download the wifi driver from the official Intel website.  Extract it somewhere (new_driver). Then:
$ls /usr/lib/firmware | grep -i pnvm
$mv *.pnvm ~/wifi/old
$mv *.ucode ~/wifi/old
$cd new_driver
sudo cp iwlwifi-*.ucode /lib/firmware/

Then reboot.

 3) Aside from this, there is a number of other things to do to try to improve the life of your adapter. I'm listing them below, I don't know which help and which don't I just did everything.

$sudo iw reg set BG #where BG is the code of YOUR country
$sudo nano /etc/modprobe.d/iwlwifi.conf
>
options iwlwifi 11n_disable=1
options iwlwifi 11n_disable=8
optwion iwlwifi wd_disable=1
ctrl+o
$sudo iw wlan0 set power_save off #this didn't work for me
$sudo iw wlp3s0 set power_save off
$sudo nano /etc/modprobe.d/wifi.conf 

> 

options iwlmvm power_scheme=1
options iwlwifi swcrypto=0 bt_coex_active=0 power_save=0 11n_disable=1 

ctrl+o

$sudo nano /etc/modprobe.d/iwlmvm.conf: 

>

options iwlmvm power_scheme=1

Key websites if you want to dig in:

https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi

https://itslinuxfoss.com/install-intel-wifi-6-ax200-driver/
https://www.intel.com/content/www/us/en/support/articles/000005511/wireless.html.

Also see this: https://bbs.archlinux.org/viewtopic.php?id=243671

But here after modeprobing the device was not visible in ifconfig and I could not try to "up" it. (also, ifconfig now is " ip a" )

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1848921

https://bbs.archlinux.org/viewtopic.php?id=243671

https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi

https://askubuntu.com/questions/1205996/iwlwifi-causing-system-hangs-on-ubuntu-19-10

https://launchpad.net/ubuntu/+source/linux-firmware/1.143.1

https://askubuntu.com/questions/616119/unstable-wireless-with-intel-7260-iwlwifi-after-upgrade-to-15-04

How to remove service from systemctl

 I had a weird service that was dumping errors like crazy in my logs visible with:

journalctl -b-1 -p4 --no-pager

So to remove it, first I found it's still running with (see this page for details):

 systemctl --type=service

then I followed these instructions and did:

systemctl stop [servicename]
systemctl disable [servicename]
rm /etc/systemd/system/[servicename]
rm /etc/systemd/system/[servicename] # and symlinks that might be related
rm /usr/lib/systemd/system/[servicename] 
rm /usr/lib/systemd/system/[servicename] # and symlinks that might be related
systemctl daemon-reload
systemctl reset-failed
 

where I basically did just the first "rm" part and the last 2 commands since the service wasn't active.

How to install Zoom virtual background on Linux

1. Install newest version of Zoom (link) 

2. Install unity-tweak-tool (I'm not sure if this is really necessairy but it started working after I installe it) 

3. Install Zoom Media plugin (link), I clicked on Citrix and Ubuntu because this is for my Mint laptop. I'll try tomorrow also on the Manjaro one. 

4, Find your ZoomMedia.ini in /etc/zoomvdi (Note, this appears after you install 3.)

5. Edit it to turn SMARTVB=0 to SMARTVB=1

To change the background, right click on the screen of the Zoom linux client, because of course, there's no simply Option to do it otherwise. But it works. Even if it took me 1 hour to figure it out.

How to kill a zombi plasmashell

 I had a case of plasmashell turning into a zombi. This is what I did to solve it:

The correct way to restart plasmashell is trough:   "plasmashell --replace &"

However, the old plasmashell was a zombi for some reason so I had to first kill it with:
 

 523  sudo killall plasmashell
 524  sudo kill {pid of the proces}
 525  ps aux | grep plasmashell
 526  cat /proc/{pid of the process}/status
 527  sudo kill {pid} 
 531  ps o ppid {pid}
 533  ps aux | grep -w Z   #shows zombi processes
 535  ps o ppid {pid}
 536  sudo kill -1 {pid}  
 539  grep plasma*
# 540  killall plasmashell && kstart5 plasmashell& #this seems to be the old way to do it

Then Alt+F2 and
plasmashell --replace &

Also see this link.

How to install polychord with conda

 That was a long battle, but it seems now all is fine. I'm using Manjaro, since Sabayon has no longer installation images with new kernels and Mocaccino is still not ready. But it seems Arch and Gentoo are not sooo different. Anyway, back to python. So this was an epic fight including downgrading binutils (of course) and installing pretty much any gfortran package out there. Here's some of what I did:

conda create --name MyEnv
conda activate MyEnv
conda install -n MyEnv pip
cd anaconda3/bin
source activate MyEnv
pip install https://github.com/PolyChord/PolyChordLite/archive/master.zip
conda install -c anaconda ipykernel
python3 -m ipykernel install --user --name=MyEnv
pip install emcee
pip install pil
pip install matplotlib
pip install arviz
pip install getdist

Alternative installation without conda:

pip install notebook

#jupyter notebook #to start it

conda deactivate

conda config --set auto_activate_base false #to deactivate conda permanently

pip install https://github.com/PolyChord/PolyChordLite/archive/master.zip

pip install build (or anything that is missing)

also don't forget to install gcc and gcc++ as they are critical for the successful installation. 

 

Below is what is currently installed in this conda env. I have no idea what I have in the base conda and in pip in general as I pretty much installed anything:

conda list --explicit
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
@EXPLICIT
https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
https://conda.anaconda.org/anaconda/linux-64/ca-certificates-2020.10.14-0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.35.1-hea4e1c9_2.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-9.3.0-h6de172a_19.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/tzdata-2021a-he74cb21_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libgomp-9.3.0-h2828fa1_19.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-1_gnu.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-9.3.0-h2828fa1_19.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libffi-3.3-h58526e2_2.tar.bz2
https://conda.anaconda.org/anaconda/linux-64/libsodium-1.0.18-h7b6447c_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.2-h58526e2_4.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/openssl-1.1.1k-h7f98852_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.5-h516909a_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.11-h516909a_1010.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/readline-8.1-h46c0cb4_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.10-h21135ba_1.tar.bz2
https://conda.anaconda.org/anaconda/linux-64/zeromq-4.3.3-he6710b0_3.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.35.5-h74cdb3f_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/python-3.9.2-hffdb5ce_0_cpython.tar.bz2
https://conda.anaconda.org/anaconda/noarch/backcall-0.2.0-py_0.tar.bz2
https://conda.anaconda.org/anaconda/noarch/decorator-4.4.2-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/ipython_genutils-0.2.0-py_1.tar.bz2
https://conda.anaconda.org/anaconda/noarch/parso-0.8.0-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pickleshare-0.7.5-py_1003.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/ptyprocess-0.7.0-pyhd3deb0d_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.9-1_cp39.tar.bz2
https://conda.anaconda.org/anaconda/noarch/six-1.15.0-py_0.tar.bz2
https://conda.anaconda.org/anaconda/noarch/wcwidth-0.2.5-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/wheel-0.36.2-pyhd3deb0d_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/certifi-2020.12.5-py39hf3d152e_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/jedi-0.18.0-py39hf3d152e_2.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pexpect-4.8.0-pyh9f0ad1d_2.tar.bz2
https://conda.anaconda.org/anaconda/noarch/python-dateutil-2.8.1-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/pyzmq-22.0.3-py39hea8fd45_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/tornado-6.1-py39h3811e60_1.tar.bz2
https://conda.anaconda.org/anaconda/noarch/traitlets-5.0.5-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/jupyter_core-4.7.1-py39hf3d152e_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/matplotlib-inline-0.1.2-pyhd8ed1ab_2.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/setuptools-49.6.0-py39hf3d152e_3.tar.bz2
https://conda.anaconda.org/anaconda/noarch/jupyter_client-6.1.7-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pip-21.1.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/anaconda/noarch/pygments-2.7.1-py_0.tar.bz2
https://conda.anaconda.org/anaconda/noarch/prompt-toolkit-3.0.8-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ipython-7.23.1-py39hef51801_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ipykernel-5.5.4-py39hef51801_0.tar.bz2

P.S. I was getting two types of errors: " ImportError: No module named 'commands ..." (this was related to installing os_sys which in the end, I don't think I need at all.  

The other error was "wheel error" which the new enviornment solved automatically. And the third one was that "f77: No such file or directo'ry", also some mix-up between the gfortran I had installed and the default name of the

In Ubuntu:

4  pip3 install pypolychord
  485  pip3 install getdist
  486  python3 -m pip install --upgrade pip
  487  pip3 install pil
  488  pip3 install pillow
  489  pip3 install --user pillow
  490  pip3 install getdist
  491  pip3 install statistics
  492  pip3 install sympi
  493  pip3 install sympy
  494  pip3 install os

Install github things with conda

I'm trying play with DALLE2 but Google collab doesn't have enough resources, so I'm testing if I can run it on my pc. So I had to install some stuff. Here is how you install directly from github:

1. Activate your conda environment source activate myenv

2. conda install git pip

3. pip install git+git://github.com/scrappy/scrappy@master

/one note, in order to do 2., I had to run it with sudo, otherwise it gave me "ERROR conda.core.link:_execute(699): An error occurred while uninstalling package  Rolling back transaction: done  [Errno 13] Permission denied: '~/anaconda3/envs/ML/include/curl/curl.h' -> '~/anaconda3/envs/ML/include/curl/curl.h.c~' ()" error.

error dlsym@GLIBC_2.2.5 when compiling

 This error, OF COURSE, was due to conda not using my system gcc etc, but I think it's good to have this round of misery archived in case I forget. (Also, my new favorite bash comannd is "history", lol )

So as always use (as many times as needed):

conda deactivate

In my case, it didn't deactivate the first time, instead it brought me back [base] which is the worst possible place to be :).  In fact this:

conda config --set auto_activate_base false

seems like a good solution.

Then it turns this is some kind of math library problem, so what I did is to replace

-lm

with 

-ldl -lmvec -lm

in the Makefile of what I was compiling. After that it compiled fine without further complaints.

I think at one point I also did as suggested here (which honestly I don't know if it was needed because i deactivated conda afterwards, so it could be just conda and the Makefile that did the trick):

EXPORT CFLAGS=" -g -O2 -lm -ldl -Wall -Wpointer-arith -finline-functions -ffast-math -funroll-all-loops";

which for me looked like:

bash -c " export CFLAGS=" -g -O2 -lm -ldl -Wall -Wpointer-arith -finline-functions -ffast-math -funroll-all-loops""

and (as suggested here)

ldd -r /home/denijane/anaconda3/envs/BAO/x86_64-conda-linux-gnu/sysroot/lib64/libdl.so.2


 

 

How to diff directories in Linux

Actually, in konsole, it's very easy to do diff, just write:

diff -q directory-1/ directory-2/

or to get a recursive comparison just for new files.

diff --brief --recursive --new-file dir1/ dir2/  # with long options 

or  

diff -qrN dir1/ dir2/   # with short flag aliases

 source

But I found it easier in this case to use gui, so I installed Meld. 

 sudo pacman -S meld

It can compare both files and directories and from what I see so far, it works very well. If you click on a file from directory diff, it automatically opens a window with file diff with highlights of the differences , which is super cool. Also, you can edit on place, Go to line and Find and Replace. If only it could remember your previous session it would be perfect. But it definitely works great for my current project.

jupyter won't start after system update (kf.service.services: KApplicationTrader: mimeType "x-scheme-handler/file" not found)

sudo pacman -S jupyter-notebook  

sudo pacman -S kde-applications

jupyter-notebook

As idiotic as it is, jupyter tried to start wine, after the update of the whole kde, it started ... Konqueror. Yup. Life's complicated when you update a linux system. Luckily, it can be solved also with 

 jupyter-notebook  --browser=firefox 

 but I found it too late. So anyway, now I have konqueror.

Fix external hard not mounting

lsblk
sudo ntfsfix /dev/sda1

How to export and load a list of dataframes without pickles

I had this problem yesterday with pickles from different pandas version not being compatible. It turned out to be a total disaster. You can read more about it here.  The worst is that I don't think there is anything you can do to fix it, they are just not compatible. Which was really bad for me - all my current local files are on pandas 1.2 and a file I calculated on a distant computer was with pandas 1.4. I couldn't downgrade the distant computer and I didn't want to upgrade my pandas as it would render all my pickles not working. In short - a disaster.

 At first I tried to load my pickle and turn it into a csv file:

import pickle as pkl
import pandas as pd
with open("file.pkl", "rb") as f:
    object = pkl.load(f)
    
df = pd.DataFrame(object)
df.to_csv(r'file.csv')

or the simpler:

df=pd.read_csv("file.csv")

df.to_pickle("file.pkl")


But that didn't work. Because I'm exporting a list of dataframes, when you import them as a csv with 

df = pd.read_csv('test.csv') 

you will get a nonsense. I tried a lot of different options, nothing rendered my list of dataframes usable - it always would produce an error. So an hour and a half later of googling and trial and error, I got to the final solution - just export all the csv separately. This doesn't work very well for me, because I have 9 dataframes per model, which means a lot of csv files which I want to avoid and which was exactly the reason to export to pickle on the first place. But for backwards and forward compatibility, I think the best way to go is this. Just export everything and then load it into the same structure and that's it.

#####

1. load pickle 

2. Save to csv #different csv files 

for i in range(0,9):

 ... f = open("test"+str(i)+".csv", 'a') 

... df[i].to_csv(f)

 ... f.close() 

3. Copy the files between computers

4. Read them: 

dfx=[0]*9 

for i in range(0,9): 

dfx[i]=pd.read_csv("test"+str(i)+".csv") 

Enjoy a working list of dataframes!!!!

Below are some of the places I read in my all-nighter to reach to this solution. I know I probably should have given up on that way earlier, but damn, I wanted to find an elegant solution to just convert pickles 1.2 to pickles 1.3 and vise versa. And I couldn't. I guess the only other thing I could do is to create a new conda environment, have the newer pandas there and work from there when needed. But considering how much trouble creating this working conda env was (some fortran problems, then some other problems, then some other other problems), i just couldn't be bothered with it. Even though it probably would have been easier and quicker.

Some useful info on csv files and pickles:  
(https://stackoverflow.com/questions/36519086/how-to-get-rid-of-unnamed-0-column-in-a-pandas-dataframe-read-in-from-csv-fil,
 https://docs.python.org/3/library/csv.html, 
https://www.listendata.com/2019/06/pandas-read-csv.html, 
https://www.geeksforgeeks.org/python-read-csv-using-pandas-read_csv/, 
https://e2eml.school/data_files.html, 
https://www.analyticsvidhya.com/blog/2021/08/python-tutorial-working-with-csv-file-for-data-science/,
 https://stackoverflow.com/questions/42168420/how-to-dill-pickle-to-file