Data analysis of a big storage

The boss bought a PB storage solution. The PB storage has a limited number of inodes and it is optimized for files of a certain size, let’s say 200MB. In principle it is a very nice improvement versus NFS mounted network drives, with the system redundancy (2 servers per task) a disk replacement service, and a backup plan. But I have a couple of problems with this.  First problem is that my boss doesn’t like quotas, so I need to scan the storage on a regularly basis, to see who’s having an indecent amount of space on it. The second one is the limited number of inodes. If you didn’t google it already, an inode “for the illiterate” is just a file, therefore, even if we have 10 PB of storage, we are allowed to store on it as much as inodes we have, let’s say 40 millions.

What I want then is my script to give me how many inodes we have remaining, and how many small files an user has on a specific folder. The first problem has an easy solution for the general case. The command “df -ih” will eve me the inodes (when in use), so I can from it grep by the share name and dump the result to a file on a specific format, like, for example:

echo `date +%m_%d`" "`df -ih | grep pbshare | awk '{print $2}'` \
>> inode_log.txt

The file inode_log.txt will log the current month and the day, followed by the number of inodes currently occupied by pbshare. You can try to run it also for the occupancy (df -h) or for whatever you want. Now here comes the interesting part, to really count the files per user. For that I will use a loop over the users of mygroup, provided they have a folder with the username on it, and find.  My analysis loop looks like this:

GROUPLIST=`getent group mygroup 
| sed 's/,/ /g' | sed 's/sb\:\*\:12345\://g'  `
for member in `echo $GROUPLIST`; do
   if [[ $(ls *$USER* 2>/dev/null ) ]]; then
       ## echo "there are folders"
       VAR2=$(for i in `find -name "*$USER*"`; \ 
              do find $i -type f -size -100c  2>/dev/null  ; \
              done | wc -l)
       VAR3=$(for i in `find -name "*$USER*"`; \ 
              do find $i -type f -size -250M  2>/dev/null  ; \ 
              done | wc -l)
       VAR4=$(for i in `find -name "*$USER*"`; \
              do find $i -type f -size +250M  2>/dev/null  ; \
              done | wc -l)
       VAR5=$(awk "BEGIN {print $VAR2+$VAR3+$VAR4; exit}")  
       echo $VAR1" "$VAR2" "$VAR3" "$VAR4" "$VAR5 

Explanation: the variables VARX are printed after the analysis for each user. The analysis loops over all the folders named after a specific user *$USER*. You could, however, try without the for loop, but it may not work if you have a lot of users. A simple version is:

VAR2=$(find *$USER* -type f -size -100c | wc -l)
VAR3=$(find *$USER* -type f -size -250M | wc -l)
VAR4=$(find *$USER* -type f -size +250M | wc -l)

Try it. Again, I print the values, but the values can go to a file each time the script run. And the file can be plotted. For that, you can use matlab, excel, or whatever you want. I decided to go for a web solution, so I used google charts. The result, although crude, seems to fulfill the purpose of it, that is, to display the data. Now, time to chase people and decide what to clean or what NOT to clean. Wish me luck!



exFAT support on CentOS 7

This is an easy one. It has been described already by kickass but as usual, I want to make a post about it as a personal note. This is also a diary, after all. So my yum command, as root, results on a package unavailable message. Therefore the solution for me was:

yum install -y

yum install exfat-utils fuse-exfat

It seems that the exfat utilities are available only for the nux-desktop repository. Here the stack overflow thread for all of it. After that, I unplug the exFAT hard disk, I plug it again and there it is, my lovely 8 TB “portable” Vault on my CentOS 7. After I dumped there all my OSX files. So mission accomplished. One day I need to make a post about hard disk formats, what you can plug, where, and what you get. If I were not so lazy…

Installing MyBB forum on Fedora 22


Do you remember my problems with mariadb service not starting on Fedora 22? Well, all of it was in order to install a simple forum on a local apache server. After a lot of playing around (I don’t want to tell you about the failures) I found the easiest and prettiest one. It’s called myBB. After the installation you see also that myBB is so good that it’s even more than a forum, it’s a statistical tool, a communication platform, and a rudimentary social network, since you can edit your profile by updating a picture, private message (PM) to somebody else on the forum and check the popularity of the members. After the flowers, let’s cut to the chase. You can download myBB here. Also you can wget, curl, and similar on your folder of choice, but I simply downloaded it. As I said, I have an apache server and a working phpMyAdmin installation, so I’m not going to explain you here and now how to get one.

After you have the zip ball unzipped you need to follow the installation instructions as explained. It’s a little bit confusing, since you get an “Upload” folder IN the upload folder. I renamed this one to my-forum, moved it upstairs and started directly with the installation. I quote: “To access the installer you must navigate to the install/ directory of your site”. That is: http://my-machine-name/my-forum/install/ or http://my-machine-IP/my-forum/install/, being my-machine-IP a number like 192.168.XX.XX or similar. You know what’s an IP, otherwise you’re not here.

Well, that’s it. You follow the installation instruction, and at the end, you’ll get a myBB forum. If I need to comment something, is about how easy is to configure it. And it gives you what it promises: a fully functional forum. You can even change the logo so it’s not a advertisement of the company. So one point less on my long, never-ending, TODO list. Have a nice weekend, all 🙂

mariadb service not starting on Fedora 22

Today I was trying to install natika on my apache server. My server is a Fedora 22 machine. I know, it’s an old system, but if it works, why to change it? It’s a rhetorical question, of course. I know if I keep it like this for a longer time, nothing will run on it anymore. Anyway, here I am that I try to install natika via composer, and I don’t happen to have composer. You can easily fix that easily by dfn (or yum) install composer. This, in my case, installed on my server like 20 php packages and updates, so I’m not surprise the installation fails when I try

composer create-project asika/natika natika *

The interesting part is why it fails, and how to fix it. The error is on the interaction between the installer and the local mysql installation. More specifically, the error reads something similar to this one:

ERROR 1045 (28000): Access denied for user 
'root'@'localhost' (using password: NO)

Now, this is one of the typical errors of mysql, when you do a careless install. Hell, I don’t remember when I did the install. It could be I did it 4 years ago. Am I careless? I could always log in on my local phpMyAdmin, so I didn’t care so much. Also

mysql -u root -p

seemed to work. Provided you give the right password. Since I installed just above a lot of new packages I decide to reset the root password using mysqld_safe:

systemctl stop mariadb.service
mysqld_safe --skip-grant-tables

and now in a new terminal

> update user set password=password('newpassword') 
where user='root'

Now I closed the first terminal and I try to restart mariadb. You can read the result by yourself.

systemctl start mariadb.service
Job for mariadb.service failed. 
See "systemctl status mariadb.service" and
"journalctl -xe" for details.
## > systemctl status mariadb.service
● mariadb.service - MariaDB 10.0 database server
 Loaded: loaded (/usr/lib/systemd/system/mariadb.service; 
enabled; vendor preset: disabled)
 Active: failed (Result: exit-code) since 11s ago
 Process: 9031 ExecStopPost=/usr/libexec/mysql-wait-stop 
(code=exited, status=3)
 Process: 8915 ExecStart=/usr/bin/mysqld_safe --basedir=/usr 
(code=exited, status=1/FAILURE)
 Main PID: 8915 (code=exited, status=1/FAILURE)

 systemd[1]: Starting MariaDB 10.0 database server...
 mysql-check-socket[8855]: Socket file /var/lib/mysql/mysql.....
 mysqld_safe[8915]: 170921 13:02:09 mysqld_safe Logging to 
 mysqld_safe[8915]: 170921 13:02:09 mysqld_safe A mysqld ...

What is wrong here? Simple, the mysql_safe process didn’t stop when I closed the first terminal. Unfortunately, a PID kill doesn’t seem to be enough. We need to stop it using mysql commands. Again Stackoverflow provide us with the solution:

mysqladmin -u root -p shutdown

And now mariadb starts cleanly. Also, composer runs flawlessly. If I find something else interesting on that field, I will let you know as usual 🙂

Not advancing

I’m not getting anywhere. My muse is gone, or on a sick leave.

I’m blocked trying to do something like google spreadsheets without google. I’d like to have it for the intranet, and I’d like to control it all. I know it’s not simple, but I can’t simply get over it. My next project is to have a working snapshot system, and for that, I also don’t have the mood. Don’t worry, if I start it, I will post my adventures here. On the literary side, I don’t want to compile my web pieces to make an ebook. I know I need a lot of time to rewrite them, to give them some continuity and clean up the edges. Hell, I don’t even want to migrate my data from my (old) MacBook Pro to my new, dark one, with a fancy touchbar. I cleaned up my office, threw away like 50 CDs of (now) useless drivers and documentation, reorganized my documents…but still, I can’t find her. Maybe she left me because I’ve not been diligent the last week. I’ve been easy-posting, and that is wrong. I need to remember to force myself to write a little each day, regarding of the quality, to keep on going.  Later I can decide what to post. Dear muse, where are thou?

100 Billion days & 100 Billion nights


The cyborg philosopher tilted his head, frowning, then began hesitantly.  “A long time ago,  I heard of a civilization that believed in an evil supreme being who existed in opposition to their good god. I have visited the village where they believed in a god of night in opposition to the god of day. The good god is the sun god, and the sun itself, and eventually the god who is in heaven – the heavenly father. I always thought that the evil god, the night god, was a purely allegorical being.”

Fragment from the book of Ryu Mitsuse, with the title above. Translated by Alexander O. Smith.  Ed. Haikasuru “The future is Japanese”

Fedora 22 sssd failed with status 3

Oh, this is a good one. I updated my apache server from Fedora 21 to Fedora 22 and I gave up on sssd because I didn’t need it at that moment. I’m the only one accessing to it, and I do it as root. Now I need it to have my AD authentication plugin working. If the machine doesn’t know who you are, how is the apache server supposed to know it? Symptoms:

> service sssd start
Redirecting to /bin/systemctl start sssd.service
Job for sssd.service failed. See "systemctl 
status sssd.service" and "journalctl -xe" for details.
> service sssd status
Redirecting to /bin/systemctl status sssd.service
● sssd.service - System Security Services Daemon
 Loaded: loaded (/usr/lib/systemd/system/sssd.service; 
disabled; vendor preset: disabled)
 Drop-In: /etc/systemd/system/sssd.service.d
 Active: failed (Result: exit-code) since "date"; 3s ago
 Process: 15032 ExecStart=/usr/sbin/sssd -D -f 
(code=exited, status=3)

(date) systemd[1]: Starting System Security Services Daemon...
(date) sssd[15034]: Starting up
(date) systemd[1]: sssd.service: 
control process exited, code=exited status=3
(date) systemd[1]: Failed to start System Security 
Services Daemon.
(date) systemd[1]: Unit sssd.service entered failed state.
(date) systemd[1]: sssd.service failed.

First let’s understand the error level. Fedora people has a SSSD troubleshooting page. They give me hope, since status=3 means:

3 Minor failures. These are the errors that would percolate down to cause the operation failure of 2

So what to do? This is what I did that didn’t work:

  • I start and stop the sssd service
  • I checked the /etc/sssd/ and /etc/krb5* permissions against a working machine
  • I removed and copy sssd.conf from another client, adapt it and restart the ssd

The answer to my problem was on the logs. Which ones? On the entry of today for more /var/log/sssd/sssd.log I see the next:

(date) [sssd] [commence_upgrade] 
(date) [sssd] [ldb] (0x0010): 
ltdb: tdb(/var/lib/sss/db/cache*): 
tdb_rec_read bad magic 0xd9fee666 at offset=1421676

Now if you google it you can get to this support page from novell that suggest to remove the database entry. Therefore I do:

  1. Stop the sssd service
  2. delete the local cache files: rm /var/lib/sss/db/*
  3. Restart the service

And… problem solved! Now to work on my plugin 😦