Find dependency packages in CLI

So I managed to compile a little GTK program!

Wow, how did 200 lines of code become a 166 KB application, let me get the dependencies so I can make an HPKG …

 objdump -x ObjectiveOasis | grep NEEDED
  NEEDED               libgtkmm-3.0.so.1
  NEEDED               libatkmm-1.6.so.1
  NEEDED               libgdkmm-3.0.so.1
  NEEDED               libgiomm-2.4.so.1
  NEEDED               libgtk-3.so.0
  NEEDED               libgdk-3.so.0
  NEEDED               libatk-1.0.so.0
  NEEDED               libcairo-gobject.so.2
  NEEDED               libgio-2.0.so.0
  NEEDED               libpangomm-1.4.so.1
  NEEDED               libglibmm-2.4.so.1
  NEEDED               libcairomm-1.0.so.1
  NEEDED               libsigc-2.0.so.0
  NEEDED               libpangocairo-1.0.so.0
  NEEDED               libpango-1.0.so.0
  NEEDED               libharfbuzz.so.0
  NEEDED               libcairo.so.2
  NEEDED               libgdk_pixbuf-2.0.so.0
  NEEDED               libgobject-2.0.so.0
  NEEDED               libglib-2.0.so.0
  NEEDED               libintl.so.8
  NEEDED               libstdc++.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libroot.so

Holy macaroni, that’s a lot of dependencies! Let’s just clean that up a little:

 objdump -x ObjectiveOasis |grep NEEDED | cut -d " " -f 18 - 
libgtkmm-3.0.so.1
libatkmm-1.6.so.1
libgdkmm-3.0.so.1
libgiomm-2.4.so.1
libgtk-3.so.0
libgdk-3.so.0
libatk-1.0.so.0
libcairo-gobject.so.2
libgio-2.0.so.0
libpangomm-1.4.so.1
libglibmm-2.4.so.1
libcairomm-1.0.so.1
libsigc-2.0.so.0
libpangocairo-1.0.so.0
libpango-1.0.so.0
libharfbuzz.so.0
libcairo.so.2
libgdk_pixbuf-2.0.so.0
libgobject-2.0.so.0
libglib-2.0.so.0
libintl.so.8
libstdc++.so.6
libgcc_s.so.1
libroot.so

At this stage, I know what to do: Use Tracker or Deskbar to Find each file, then use @humdinger’s excellent openoriginpackageTracker add-on to find out which package gifted me with this library. And that is fine when there are three or four dependencies. But when the list starts to look like this, well …

So the question is, is there a command line equivalent to the Tracker add-on? Something that can be put in a script? I’ve tried pkgman search. It works on some, but not on others.

2 Likes

All files from a package have an attribute SYS:PACKAGE_FILE that points to its package. So, once you have a list of packages, use catattr SYS:PACKAGE_FILE on each.

4 Likes

Don’t do that? Packages providing libraries should declare them, you should require lib:libgtkmm_3.0 instead of gtkmm3.

So:
objdump -x ObjectiveOasis | sed -n '/^\s*NEEDED\s/{s/^\s*NEEDED\s*\(.*\)\.so.*/lib:\1/;s/-/_/;p}'

If you are interested, you can pass that to pkgman search, but it seems it does substring comparison, so if you look for lib:foo it will also list a package providing lib:foobar:

objdump -x ObjectiveOasis | sed -n '/^\s*NEEDED\s/{s/^\s*NEEDED\s*\(.*\)\.so.*/lib:\1/;s/-/_/;p}' | while read -r line; do echo $line; pkgman search "${line}" | tail -n +3; done

Or, if you want to know the packages from where the files you have with that name came from:

catattr -d 'SYS:PACKAGE' $(objdump -x ObjectiveOasis | sed -n 's/^\s*NEEDED\s*//p' | while read -r line; do query -a name="${line}"; done) 2> /dev/null | sort | uniq

You may prefer to not redirect error output, in case some of the libraries come from development versions without the attribute. And you may want to only search under /boot/system.

2 Likes

Thanks, @humdinger. That was just what I was looking for. Many hours and many manpages later. I’ve cobbled together something that works for me. Probably Almost definitely not bulletproof, but I’ve tested it against some files with known dependencies and it seems to work:

findpkgs

#!/bin/sh
if [ $# -eq 0 ]; then
	echo "Usage: findpkgs <executable_filename>"
	echo "       and no spaces in the filename, please."
	exit 1
fi
echo Processing ...
updatedb &
tmpfile1=$(mktemp)
tmpfile2=$(mktemp)
tmpfile3=$(mktemp)
objdump -x $1 |grep NEEDED | cut -d " " -f 18 - > $tmpfile1
for i in $(cat $tmpfile1); do
	locate -b -e -l 1 $i | sort |grep /boot/system/develop/lib >> $tmpfile2
	locate -b -e -l 1 $i | sort | grep /boot/system/lib/ >> $tmpfile2
done
for i in $(cat $tmpfile2); do
	catattr -d SYS:PACKAGE_FILE $i >> $tmpfile3 
done 
cat $tmpfile3 | uniq
rm -f $tmpfile1 $tmpfile2 $tmpfile3
exit 0
2 Likes

Thanks, @madmax. I’m going to have to brush up on my sed skills and get back to you.

Thanks! Gave it a quick check and output looked what I was hoping for. +1

Let’s split those for explanations.

sed -n '/^\s*NEEDED\s/{s/^\s*NEEDED\s*\(.*\)\.so.*/lib:\1/;s/-/_/;p}:

-n                      # no automatic print
/regex/ {...}           # apply block to lines selected by /regex/:
    ^                   # anchor at the beginning
    \s*                 # any number of whitespaces
    NEEDED              # literal string
    \s                  # one whitespace

s/X/Y/                  # substitute X for Y
                        # it will match the whole line,
                        # save whatever is between the spaces and '.so',
                        # and prepend 'lib:' to the saved part
  X:
    ^                   # anchor at the beginning
    \s*                 # any number of whitespaces
    NEEDED              # literal string
    \s*                 # any number of whitespaces
    \(                  # start of capture group
    .*                  # any number of characters
    \)                  # end of capture group
    \.                  # literal dot
    so                  # literal string
    .*                  # any number of characters
  Y:
    lib:                # literal lib:
    \1                  # the first capture group
;                       # command separator
s/-/_/;                 # substitute '-' for '_'
                        # BUG ALERT!!!
                        # This only substitutes the first match. It should be:
                        # y/-/_/;
p                       # print

The other one, sed -n 's/^\s*NEEDED\s*//p' just selects the NEEDED lines and removes that part (substitutes it for nothing), leaving just the library name. Like your grep and cut, but in one process and without having to count spaces.

May I also hijack the thread for some tips and transform your script into something like my one-liner?

objdump -x $1 |grep NEEDED | cut -d " " -f 18 - > $tmpfile1

You can tell by now that I prefer not to count and use one program instead of two:

objdump -x "$1" | sed -n 's/^\s*NEEDED\s*//p' > "${tmpfile1}"

Also notice argument quoting. That allows us to take a filename with spaces in it.

for i in $(cat $tmpfile1); do
	locate -b -e -l 1 $i | sort |grep /boot/system/develop/lib >> $tmpfile2
	locate -b -e -l 1 $i | sort | grep /boot/system/lib/ >> $tmpfile2
done

You are telling locate to only output one match with -l 1, so there’s no point in sorting it. Even if you weren’t, it’s better to sort after grep: there’d be less stuff to sort. In fact, you don’t want to limit it, as the first match may not be in the directories you want.

You can pass several filters to grep, halving the processes launched. You can also build a regex to match everything, but let’s leave that for another life. You’ll also want to anchor the match to the beginning of the line, just in case you have those as part of a deep path somewhere.

for i in $(cat "${tmpfile1}"); do
	locate -b -e $i | grep -e '^/boot/system/develop/lib' -e '^/boot/system/lib/' >> "${tmpfile2}"
done

I’ve left sort out of here for the moment. There’s no point in sorting the results for each of the libraries.

You can further reduce the number of processes by grepping the result of the whole loop instead of every iteration:

for i in $(cat $tmpfile1); do
	locate -b -e $i
done | grep -e '^/boot/system/develop/lib' -e '^/boot/system/lib/' >> $tmpfile2

Extreme example of impact:

> time for f in *; do echo $f; done | grep -e w -e x
...
real	0m0,006s
user	0m0,006s
sys	0m0,003s

> time for f in *; do echo $f | sort | grep w; echo $f | sort | grep x; done
...
real	0m0,369s
user	0m0,139s
sys	0m0,448s

You may even be able to directly tell locate to search in those directories. But then I’ve not used locate or find in Haiku to look for full filenames since I learnt about query. Why keep a database updated or traverse the filesystem when it already has an index? So let’s change locate -b -e $i into query -a name="$i" and get rid of updatedb, which, by the way you left running in the background, so those locate processes may or not get data from an updated database. Quoting (notice I used "$i" instead of $i) will help with “spaces in the filenames of results” problems. Not yet, due to how you are looping, but we’ll get to that.

cat $tmpfile3 | uniq

Even with your previous sorts, the data in $tmpfile is not sorted, and duplicates may be sprinkled all around:

cat $tmpfile3 | sort | uniq

Speaking of which, all those dumping to a file to just cat it and delete it are useless in the final script. As you are not reading from them several times or anything like that, you can avoid creating a file, writing to it, launching a process and reading from it. For example, instead of splitting the output of cat $tmpfile1 and looping it, just do so with the output of objdump:

for i in $(objdump -x "$1" | sed -n 's/^\s*NEEDED\s*//p'); do

Better yet, pipe it and read it line by line, thus not splitting by space.

objdump -x "$1" | sed -n 's/^\s*NEEDED\s*//p' | while read -r i; do

You could also change the IFS environment variable, but I won’t try that, although we could use it to launch just one catattr.

After all of that, we get to this script:

#!/bin/sh
if [ $# -ne 1 -o "$1" == "--help" ]; then
	echo "Usage: findpkgs <executable_filename>"
	echo "       and no newlines in the filenames, please."
	exit 1
fi

objdump -x "$1" | sed -n 's/^\s*NEEDED\s*//p' | while read -r library; do
	query -a name="${library}"
done | grep -e '^/boot/system/develop/lib' -e '^/boot/system/lib/' | sort | uniq | while read -r file; do
	catattr -d SYS:PACKAGE_FILE "${file}"
done | sort | uniq
6 Likes

Just noticed sort can remove duplicates. Avoid piping to uniq by using sort -u.

1 Like

Ahoy @madmax,

Thanks for your tips and hints doing efficient shell scripting lessons - it was interesting to follow the examples …
I had not cared about the possibilities so deeply when I had written my scripts, I just used the commands and piping like legos in the past - I should have trained to take care efficiency as well :smiley:

Ahoy @michel ,

Aye, not bulletproof … I assume you need it for development, so you are in the directory where the executable(s) is(are) … so you had not faced with to locate the exec file while using the script :smiley:

~> ls -l findpkgs 
-rw-r--r-- 1 user root 428 febr.   9 12:49 findpkgs
~> chmod 755 findpkgs
~> findpkgs
Usage: findpkgs <executable_filename>
       and no newlines in the filenames, please.
~> findpkgs wine_bin
objdump: 'wine_bin': No such file
~> findpkgs boxedwine
objdump: 'boxedwine': No such file
~> find /boot -name boxedwine
~> find /boot -name Boxedwine
/boot/home/config/settings/Boxedwine
/boot/system/data/deskbar/menu/Applications/Boxedwine
/boot/system/apps/Boxedwine
~> findpkgs Boxedwine
objdump: 'Boxedwine': No such file
~> findpkgs /boot/system/apps/Boxedwine
gcc_syslibs-13.3.0_2023_08_10-4-x86_64.hpkg
glew-2.2.0-3-x86_64.hpkg
glu-9.0.0-8-x86_64.hpkg
haiku-r1~beta5_hrev58610-1-x86_64.hpkg
libsdl2-2.30.10-1-x86_64.hpkg
mesa-22.0.5-3-x86_64.hpkg
minizip-1.3-1-x86_64.hpkg
openssl-1.1.1w-2-x86_64.hpkg
zlib-1.3.1-4-x86_64.hpkg
~> 
~> cat findpkgs
#!/bin/sh
if [ $# -ne 1 -o "$1" == "--help" ]; then
        echo "Usage: findpkgs <executable_filename>"
        echo "       and no newlines in the filenames, please."
        exit 1
fi

objdump -x "$1" | sed -n 's/^\s*NEEDED\s*//p' | while read -r library; do
        query -a name="${library}"
done | grep -e '^/boot/system/develop/lib' -e '^/boot/system/lib/' | sort | uniq | while read -r file; do
        catattr -d SYS:PACKAGE_FILE "${file}"
done | sort -u
~> 

EDIT :

also interesting that package name and binaries are different, some of them so like their directories not contained in the path of binaries, so when I try to launch them in Terminal … better to check them : on which name and from where I can launch them.
The best was bootmanager - it was just in Installer’s menu to launch it … none of the videos or hints were not about it. I had found it recently … by accidentally (after it was mentioned in a doc - I can find it only this way !).