Tux

...making Linux just a little more fun!

Talkback:164/lg_tips.html#1

[ In reference to "2-Cent Tips" in LG#164 ]

Paul Sephton [paul at inet.co.za]


Fri, 03 Jul 2009 23:14:47 +0200

@$# This was originally entitled: "about 2c ext2 fragmentation" - Kat $#@

On Fri, 2009-07-03 at 22:46 +0200, Carlos Baiget wrote:

> another 0.5c: If any file in the given directory has a space in his 
> name, the defragmentation script will not work. To replace all spaces 
> inside the file name, i suggest to do a
> 
>  rename 's/\ /_/g' *
> 
> previously or modify IFS environment variable to not consider space as 
> field separator.
> 
> Carlos

Ah, yes; or simply quote the file names as follows:

    fs=`echo "$line" | cut -f 1 -d':'`
    fn=`echo "$line" | cut -f 2 -d':'`
# copy the file up to 10 times, preserving permissions
    j=0;
    while [ -f "$fn" -a $j -lt 10 ]; do
    ....

IFS also makes sense. Thanks for highlighting this rather serious error. I really should have taken the trouble to test this properly first!

Paul


Top    Back


Paul Sephton [paul at inet.co.za]


Fri, 03 Jul 2009 23:17:55 +0200

Aaaahhhh! Top posted! Too late, she cried!

[[[ It's okay, Paul - for you, I manually corrected it. Thanks for noticing and caring. -- Kat ]]]


Top    Back


Thomas Adam [thomas.adam22 at gmail.com]


Fri, 3 Jul 2009 22:24:27 +0100

2009/7/3 Paul Sephton <[email protected]>:

> Ah, yes;  or simply quote the file names as follows:
>
>    fs=`echo "$line" | cut -f 1 -d':'`
>    fn=`echo "$line" | cut -f 2 -d':'`
> # copy the file up to 10 times, preserving permissions
>    j=0;
>    while [ -f "$fn" -a $j -lt 10 ]; do
>    ....
>
> IFS also makes sense.  Thanks for highlighting this rather serious error.  I
> really should have taken the trouble to test this properly first!
>
> Paul
>
> On Fri, 2009-07-03 at 22:46 +0200, Carlos Baiget wrote:
>> another 0.5c: If any file in the given directory has a space in his
>> name, the defragmentation script will not work. To replace all spaces
>> inside the file name, i suggest to do a
>>
>>  rename 's/\ /_/g' *

Err, be careful here -- rename as a command is not the same across distributions. On Debian and I assume it's derivatives, it is indeed the classic perl script. On Redhat, it's something completely different. In which case "mmv" and friends can be used.

-- Thomas Adam


Top    Back


Paul Sephton [paul at inet.co.za]


Fri, 03 Jul 2009 23:52:26 +0200

On Fri, 2009-07-03 at 22:24 +0100, Thomas Adam wrote:

> 2009/7/3 Paul Sephton <[email protected]>:
> > On Fri, 2009-07-03 at 22:46 +0200, Carlos Baiget wrote:
> >> another 0.5c: If any file in the given directory has a space in his
> >> name, the defragmentation script will not work. To replace all spaces
> >> inside the file name, i suggest to do a
> >>
> >>  rename 's/\ /_/g' *
> 
> Err, be careful here -- rename as a command is not the same across
> distributions.  On Debian and I assume it's derivatives, it is indeed
> the classic perl script.  On Redhat, it's something completely
> different.  In which case "mmv" and friends can be used.
> 
> -- Thomas Adam

Good point. Anyway, the other problem with the script is that it needs root privilege to run '/sbin/filefrag'. The script is not really safe either as pointed out by Carlos. I have tested the following against a directory of files that contain spaces. On the other hand, the fact that 'filefrag' can't be run as a normal user rather limits it's use. chmod +s /sbin/filefrag is an option.

Can you spot any more loopholes?

#!/bin/sh
# Retrieve a list for fragmented files, #fragments:filename
FILEFRAG="/sbin/filefrag"
if [ ! -f $FILEFRAG ]; then
  echo requires $FILEFRAG to defrag this directory
  exit 0
fi
 
flist() {
  for i in *; do
    if [ -f "$i" ]; then
      ff=`$FILEFRAG "$i"`
      fn=`echo "$ff" | cut -f1 -d':'`
      fs=`echo "$ff" | cut -f2 -d':' | cut -f2 -d' '`
      if [ -f "$fn" -a $fs -gt 1 ]; then echo -e "$fs:$fn"; fi
    fi
  done
}
IFS='
'
 
# Sort the list numeric, descending
flist | sort -n -r |
(
# for each file
  while read line; do
    fs=`echo "$line" | cut -f 1 -d':'`
    fn=`echo "$line" | cut -f 2 -d':'`
# copy the file up to 10 times, preserving permissions
    j=0;
    while [ -f "$fn" -a $j -lt 10 ]; do
      j=$[ $j + 1 ]
 
      TMP=$$.tmp.$j
      if ! cp -p "$fn" "$TMP"; then
        echo copy failed [$fn]
        j=10
      else
# test the new temp file's fragmentation, and if less than the
# original, move the temp file over the original
        ns=`$FILEFRAG $TMP | cut -f2 -d':' | cut -f2 -d' '`
        if [ $ns -lt $fs ]; then
          mv "$TMP" "$fn"
          fs=$ns
          if [ $ns -lt 2 ]; then j=10; fi
        fi
      fi
    done
    j=0;
# clean up temporary files
    while [ $j -lt 10 ]; do
      j=$[ $j + 1 ]
 
      TMP=$$.tmp.$j
      if [ -f "$TMP" ]; then
        rm "$TMP"
      else
        j=10
      fi
    done
  done
)
# report fragmentation
for i in *; do if [ -f $i ]; then $FILEFRAG $i; fi; done


Top    Back


Thomas Adam [thomas.adam22 at gmail.com]


Fri, 3 Jul 2009 22:54:40 +0100

2009/7/3 Paul Sephton <[email protected]>:

> Can you spot any more loopholes?

I scanned it very quickly, you, like everyone else, needs to "USE" ""MORE"" """""QUOTES""""".

I can't stress enough just how important that is.

-- Thomas Adam


Top    Back


Paul Sephton [paul at inet.co.za]


Sat, 04 Jul 2009 00:02:26 +0200

On Fri, 2009-07-03 at 22:54 +0100, Thomas Adam wrote:

> 2009/7/3 Paul Sephton <[email protected]>:
> > Can you spot any more loopholes?
> 
> I scanned it very quickly, you, like everyone else, needs to "USE"
> ""MORE"" """""QUOTES""""".
> 
> I can't stress enough just how important that is.
> 
> -- Thomas Adam

... as in the last line,

   for i in *; do if [ -f $i ]; then $FILEFRAG $i; fi; done
should be
   for i in *; do if [ -f "$i" ]; then $FILEFRAG "$i"; fi; done

oh well....


Top    Back


Paul Sephton [paul at inet.co.za]


Sat, 04 Jul 2009 00:42:41 +0200

On Fri, 2009-07-03 at 22:54 +0100, Thomas Adam wrote:

> 2009/7/3 Paul Sephton <[email protected]>:
> > Can you spot any more loopholes?
> 
> I scanned it very quickly, you, like everyone else, needs to "USE"
> ""MORE"" """""QUOTES""""".
> I can't stress enough just how important that is.
> -- Thomas Adam

Ok, here's the latest. Gives a bit more feedback about what is happening in a more readable way. Also fixes a bug that left temp files lying around. I'm running out of still fragmented directories/files to test it on.

#!/bin/sh
# Retrieve a list for fragmented files, #fragments:filename
FILEFRAG="/sbin/filefrag"
if [ ! -f $FILEFRAG ]; then
  echo requires $FILEFRAG to defrag this directory
  exit 0
fi
 
flist() {
  for i in *; do
    if [ -f "$i" ]; then
      ff=`$FILEFRAG "$i"`
      fn=`echo "$ff" | cut -f1 -d':'`
      fs=`echo "$ff" | cut -f2 -d':' | cut -f2 -d' '`
      if [ -f "$fn" -a $fs -gt 1 ]; then echo -e "$fs:$fn"; fi
    fi
  done
}
 
# Internal Field Separator is now a newline
IFS='
'
 
# Sort the list numeric, descending
echo Reading files...
flist | sort -n -r |
(
# for each file
  while read line; do
    fs=`echo "$line" | cut -f 1 -d':'`
    fn=`echo "$line" | cut -f 2 -d':'`
# copy the file up to 10 times, preserving permissions
    j=0;
    echo -e -n "$fn\t$fs"
    while [ -f "$fn" -a $j -lt 10 ]; do
      j=$[ $j + 1 ]
      echo -n "."
      TMP=$$.tmp.$j
      if ! cp -p "$fn" "$TMP"; then
        echo copy failed [$fn]
        j=10
      else
# test the new temp file's fragmentation, and if less than the
# original, move the temp file over the original
        ns=`$FILEFRAG $TMP | cut -f2 -d':' | cut -f2 -d' '`
        if [ $ns -lt $fs ]; then
          mv "$TMP" "$fn"
          fs=$ns
    	  echo -n "$fs"
          if [ $ns -lt 2 ]; then j=10; fi
        fi
      fi
    done
    echo
    echo "    - file now has $fs fragments"
    j=0;
# clean up temporary files
    while [ $j -lt 10 ]; do
      j=$[ $j + 1 ]
 
      TMP=$$.tmp.$j
      if [ -f "$TMP" ]; then
        rm "$TMP"
      fi
    done
  done
)


Top    Back


Thomas Adam [thomas.adam22 at gmail.com]


Fri, 3 Jul 2009 23:50:24 +0100

2009/7/3 Paul Sephton <[email protected]>:

> if [ ! -f $FILEFRAG ]; then

[ -x "$FILEFRAG" ]

Just on the off-chance the file isn't executable for some odd reason.

>  echo requires $FILEFRAG to defrag this directory

echo "requires $FILEFRAG to defrag this directory"
>  exit 0
> fi
>
> flist() {
>  for i in *; do
>    if [ -f "$i" ]; then
>      ff=`$FILEFRAG "$i"`

To be honest, only the most ancient of shells, and I mean ancient, won't understand:

$()

For command substitution, over backticks. Given nesting backticks is problematic, you should just use "$()".

>      fn=`echo "$ff" | cut -f1 -d':'`
>      fs=`echo "$ff" | cut -f2 -d':' | cut -f2 -d' '`
>      if [ -f "$fn" -a $fs -gt 1 ]; then echo -e "$fs:$fn"; fi

Odd -- you claim "/bin/sh", yet "echo -e" isn't POSIX -- and why is it even invoked as such here, when you're not asking for backslash interpretation?

>    echo -e -n "$fn\t$fs"

printf()

>      j=$[ $j + 1 ]

((J++)) -- the square-brackets are an arcane syntax.

-- Thomas Adam


Top    Back


Paul Sephton [paul at inet.co.za]


Sat, 04 Jul 2009 01:12:14 +0200

On Fri, 2009-07-03 at 23:50 +0100, Thomas Adam wrote:

> 2009/7/3 Paul Sephton <[email protected]>:
> > if [ ! -f $FILEFRAG ]; then
> 
> ```
> [ -x "$FILEFRAG" ]
> '''
> 
> Just on the off-chance the file isn't executable for some odd reason.

Cool. I had changed it to -x elsewhere, but the real problem is if the script is run as an ordinary user, the execution fails because of a kernel access restriction. I was thinking of checking

  -x $FILEFRAG -a $USER=root or
  -x $FILEFRAG -a -s $FILEFRAG.  Might be better.
> >  echo requires $FILEFRAG to defrag this directory
> 
> ```
> echo "requires $FILEFRAG to defrag this directory"
> 
> >  exit 0
> > fi
> >
> > flist() {
> >  for i in *; do
> >    if [ -f "$i" ]; then
> >      ff=`$FILEFRAG "$i"`
> 
> To be honest, only the most ancient of shells, and I mean ancient,
> won't understand:
> 
> ```
> $()
> '''

... you can tell I'm rather ancient, can't you? I have never used $() before, other than in makefiles...

> For command substitution, over backticks.  Given nesting backticks is
> problematic, you should just use "$()".
> 
> >      fn=`echo "$ff" | cut -f1 -d':'`
> >      fs=`echo "$ff" | cut -f2 -d':' | cut -f2 -d' '`
> >      if [ -f "$fn" -a $fs -gt 1 ]; then echo -e "$fs:$fn"; fi
> 
> Odd -- you claim "/bin/sh", yet "echo -e" isn't POSIX -- and why is it
> even invoked as such here, when you're not asking for backslash
> interpretation?
> 
> >    echo -e -n "$fn\t$fs"
> 
> printf()

printf() is POSIX? I didn't know. Agreed, echo -e is unnecessary in the first instance. I'll just make it #bin/bash rather, or as you suggest, try printf() instead.

> 
> >      j=$[ $j + 1 ]
> 
> ((J++)) -- the square-brackets are an arcane syntax.

I'm not just ancient, but a bit arcane too :-)

> -- Thomas Adam

Thanks, Thomas. Your help is appreciated.


Top    Back


Paul Sephton [paul at inet.co.za]


Sat, 04 Jul 2009 10:38:59 +0200

On Fri, 2009-07-03 at 23:50 +0100, Thomas Adam wrote:

> 2009/7/3 Paul Sephton <[email protected]>:
> > if [ ! -f $FILEFRAG ]; then
> 
> ```
> [ -x "$FILEFRAG" ]
> '''
> 
> Just on the off-chance the file isn't executable for some odd reason.

<snip> Ok, this actually reads and works a whole lot better. Thanks for the tips, Thomas:

#!/bin/sh
# Retrieve a list for fragmented files, #fragments:filename
FILEFRAG="/sbin/filefrag"
if [ ! -x $FILEFRAG ]; then
  echo requires $FILEFRAG to defrag this directory
  exit 0
fi
if [ ! "$USER" = "root" ]; then
  if [ ! -u $FILEFRAG ]; then
    echo $FILEFRAG can only be run as root, or should be SUID root
    exit 0
  fi
fi
 
flist() {
  for i in *; do
    if [ -f "$i" ]; then
      ff=$( $FILEFRAG "$i" )
      fn=$( echo "$ff" | cut -f1 -d':' )
      fs=$( echo "$ff" | cut -f2 -d':' | cut -f2 -d' ' )
      if [ -f "$fn" -a $fs -gt 1 ]; then printf "$fs:$fn\n"; fi
    fi
  done
}
 
# Internal Field Separator is now a newline
IFS='
'
 
# Sort the list numeric, descending
echo Reading files...
flist | sort -n -r |
(
# for each file
  while read line; do
    fs=$( echo "$line" | cut -f 1 -d':' )
    fn=$( echo "$line" | cut -f 2 -d':' )
# copy the file up to 10 times, preserving permissions
    j=0
    printf "$fn\t$fs"
    while [ -f "$fn" -a $j -lt 10 ]; do
      ((j++))
      printf "."
      TMP=$$.tmp.$j
      if [ -f "$TMP" ]; then
        echo "Error! target temp file already exists!"
      elif ! cp -p "$fn" "$TMP"; then
        echo copy failed [$fn]
        j=10
      else
# test the new temp file's fragmentation, and if less than the
# original, move the temp file over the original
        ns=$( $FILEFRAG "$TMP" | cut -f2 -d':' | cut -f2 -d' ' )
        if [ $ns -lt $fs ]; then
          mv "$TMP" "$fn"
          fs=$ns
    	  printf "$fs"
          if [ $ns -lt 2 ]; then j=10; fi
        fi
      fi
    done
    echo
    echo "    (file now has $fs fragments)"
    j=0
# clean up temporary files
    while [ $j -lt 10 ]; do
      ((j++))
 
      TMP=$$.tmp.$j
      if [ -f "$TMP" ]; then
        rm "$TMP"
      fi
    done
  done
)


Top    Back