четвъртък, 9 април 2015 г.

Cool uses of sed

I really fell in love with this site: sed - 20 examples to remove / delete characters from a file . 
To remove first character only if it is a specific character:
$ sed 's/^F//' file > file.new
Pretty cool when you need to deal with 10 000 incompatible ways to comment in Fortran.
Also, one Fortran note to self:
To fix deprecated assign to and go to:

(a) change each of the ASSIGN N TO NEXT (where N is some number) statement to a simple NEXT = N statement (where N is the same number), and

(b) replace each of the GO TO NEXT, (x, y, z, ...) statement with the following computed goto statement

GO TO (1,2,3,4,5,6,7,8,9,10,11) NEXT

сряда, 1 април 2015 г.

How to limit RAM usage of Fortran

Obviously, numerical computing is a BITCH! Recently, my system went havoc because of one integration taking more than 10GB RAM. In principle I have 16GB RAM but I had tons of stuff going on and I never expected the fortran code to take so much resources. Which was kind of naive from my side, but anyway.
So here is how you can limit the memory usage of your programs.
The easiest way is:
$ ulimit -v
unlimited
$ ulimit -v 100000
$ python -c '[ "x" * 100000000 ]'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
MemoryError
This is from here.
Also, you can mess with the stacksize this way.
        ulimit -S -s
      
For me, it gives "8192". To increase it:
        ulimit -S -s 16384
      
or:
        ulimit -s unlimited
      
I would recommend against this, however, unless the program explicitly complains against the stacksize. I remember doing this for Maple and it turned out a disaster. And in Maple at least, there is another way to increase the heap size which solves most memory related problems.
Anyway, this method is not very good, because it requires constant change of the ulimit.
Another more interesting option is the use of cgroups:
$gcreate -g memory:/myGroup
$ echo $(( 500 * 1024 * 1024 )) > /sys/fs/cgroup/memory/myGroup/memory.limit_in_bytes
$ echo $(( 5000 * 1024 * 1024 )) > /sys/fs/cgroup/memory/myGroup/memory.memsw.limit_in_bytes
cgexec -g memory:myGroup pdftoppm
/source site/
Note that in order to set the limits on the swap, you need to add
GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1"
to /etc/default/grub
 For a very detailed info on cgroups you can see here.
Also in the ArcWiki (cgroups) there is a great example how to limit the memory of matlab for example. But in my case, I think it's easier to just create a group and launch my program from within it.
We'll see about that.