Add git-annex library to the package

For now abandoning the idea of using the git-annex standalone. After building the package with git-annex standalone setup the health check failed because large number of libraries still use external libraries.

To build git-annex, haskell-platform is required.

To build haskell-platform, GHC, Cabal, haddock and HsColour are required.

To build GHC, GHC is required (yes, GHC to build GHC, building prerequisites )

To build Cabal, network is required.

I went a step back to try and compile this from source by hand following the official guide and I got the failure before I could start:

cabal build

Utility/Exception.hs:25:18:
    Could not find module `Control.Monad.Catch'
    Perhaps you meant
      Control.Monad.CatchIO (from MonadCatchIO-mtl-0.3.0.4)
      Control.Monad.Base (from transformers-base-0.4.1)
      Control.Monad.Cont (from mtl-2.0.1.0)
    Use -v to see a list of the files searched for.

Progressed a bit further by installing few modules:

cabal install catch-fd
cabal install unix-compat
cabal install data-default

only to fail again:

esolving dependencies...
[ 7 of 31] Compiling Utility.Exception ( Utility/Exception.hs, dist/setup/Utility/Exception.o )

Utility/Exception.hs:24:41:
    Module `Control.Monad.Catch' does not export `Handler'
make: *** [Build/SysConfig.hs] Error 1

This is a rabbit hole, more and more modules are giving errors and I am suspecting this is due to missing packages(from cabal). Possibly outdated.

Current version of git-annex cannot be compiled against haskell libraries that come with OS repositories.

To compile current git-annex haskell-platform needs to be compiled from source.

This means that I need to compile GHC, haskell-platform and all that comes with it.

To compile GHC, I did apt-get install ghc as per building requirements .

Downloaded and unpacked,

./configure
make 
make install

Failed on ./configure due to missing ncurses lib.

Installed libncurses5-dev.

Configure failed:

checking whether bootstrap compiler is affected by bug 9439... Warning: Couldn't figure out LLVM version!
         Make sure you have installed LLVM
ghc: could not execute: opt-3.0

Had to install llvm.

Next step was to install cabal:

Download and unpacked

and then

ghc -threaded --make Setup
    ./Setup configure --user
    ./Setup build
    ./Setup install

I thought that cabal would be enough to get cabal but it seems not, I had to install cabal-install

download and unpacked

ghc-pkg check
./bootstrap.sh

Since I just compiled it without forcing it into any location in the PATH i had to add it to the path in bashrc :

CABAL_HOME=$HOME/.cabal
PATH=$CABAL_HOME/bin:$PATH

Next was haskell-platform. Downloaded and unpacked

and ran the installer ./plaform.sh

Failed with

src/OS/Win.hs:78:48:
    Ambiguous occurrence `C.installedPackageId'
    It could refer to either `C.installedPackageId',
                             imported qualified from `Distribution.InstalledPackageInfo' at src/OS/Win.hs:14:1-55
                          or `C.installedPackageId',
                             imported qualified from `Distribution.Package' at src/OS/Win.hs:15:1-42

Ran cabal install in haskell-platform source directory. This installed number of packages but it failed with

cabal: Error: some packages failed to install:
GLURaw-1.4.0.1 depends on OpenGLRaw-1.5.0.0 which failed to install.
GLUT-2.5.1.1 depends on OpenGLRaw-1.5.0.0 which failed to install.
OpenGL-2.9.2.0 depends on OpenGLRaw-1.5.0.0 which failed to install.
OpenGLRaw-1.5.0.0 failed during the configure step. The exception was:
ExitFailure 1

Installed apt-get install freeglut3 freeglut3-dev and ran cabal install OpenGLRaw-1.5.0.0 which failed again with the same error.

Found this issue https://github.com/haskell/haskell-platform/issues/164

and in with vim hptool/src/OS/Win.hs I commented out line 14.

Ran ./platform.sh again and haskell-platform finished installation.

Downloaded git-annex from http://git-annex.branchable.com/download/

and tried configuring with minimal number of flags: cabal configure -f"-assistant -pairing -xmpp -dns" and failed with

Common.hs:14:8:
    Could not find module `Data.Default'
    Use -v to see a list of the files searched for.

Utility/Exception.hs:25:18:
    Could not find module `Control.Monad.Catch'
    Perhaps you meant
      Control.Monad.Cont (from mtl-2.2.1)
      Control.Monad.Cont (needs flag -package mtl-2.1.3.1)
      Control.Monad.State (needs flag -package mtl-2.1.3.1)
    Use -v to see a list of the files searched for.

Utility/FileSize.hs:10:8:
    Could not find module `System.PosixCompat.Files'
    Perhaps you meant System.Posix.Files (from unix-2.7.0.1)
    Use -v to see a list of the files searched for.

Utility/FileSystemEncoding.hs:25:18:
    Could not find module `Data.Hash.MD5'
    Use -v to see a list of the files searched for.

Utility/FileSystemEncoding.hs:27:8:
    Could not find module `Data.Bits.Utils'
    Use -v to see a list of the files searched for.

Utility/Process.hs:45:8:
    Could not find module `System.Log.Logger'
    Use -v to see a list of the files searched for.

Utility/SafeCommand.hs:12:8:
    Could not find module `Data.String.Utils'
    Use -v to see a list of the files searched for.

Utility/Tmp.hs:14:8:
    Could not find module `Control.Monad.IfElse'
    Perhaps you meant
      Control.Monad.IO.Class (needs flag -package transformers-0.3.0.0)
      Control.Monad.List (needs flag -package mtl-2.1.3.1)
      Control.Monad.State (needs flag -package mtl-2.1.3.1)
    Use -v to see a list of the files searched for.

Utility/UserInfo.hs:16:8:
    Could not find module `System.PosixCompat'
    Perhaps you meant
      System.Posix.Temp (from unix-2.7.0.1)
      System.Posix.Types (from base)
    Use -v to see a list of the files searched for.
root@gitlab-dev:/tmp/git-annex# cabal configure
Resolving dependencies...

Common.hs:14:8:
    Could not find module `Data.Default'
    Use -v to see a list of the files searched for.

Utility/Exception.hs:25:18:
    Could not find module `Control.Monad.Catch'
    Perhaps you meant
      Control.Monad.Cont (from mtl-2.2.1)
      Control.Monad.Cont (needs flag -package mtl-2.1.3.1)
      Control.Monad.State (needs flag -package mtl-2.1.3.1)
    Use -v to see a list of the files searched for.

Utility/FileSize.hs:10:8:
    Could not find module `System.PosixCompat.Files'
    Perhaps you meant System.Posix.Files (from unix-2.7.0.1)
    Use -v to see a list of the files searched for.

Utility/FileSystemEncoding.hs:25:18:
    Could not find module `Data.Hash.MD5'
    Use -v to see a list of the files searched for.

Utility/FileSystemEncoding.hs:27:8:
    Could not find module `Data.Bits.Utils'
    Use -v to see a list of the files searched for.

Utility/Process.hs:45:8:
    Could not find module `System.Log.Logger'
    Use -v to see a list of the files searched for.

Utility/SafeCommand.hs:12:8:
    Could not find module `Data.String.Utils'
    Use -v to see a list of the files searched for.

Utility/Tmp.hs:14:8:
    Could not find module `Control.Monad.IfElse'
    Perhaps you meant
      Control.Monad.IO.Class (needs flag -package transformers-0.3.0.0)
      Control.Monad.List (needs flag -package mtl-2.1.3.1)
      Control.Monad.State (needs flag -package mtl-2.1.3.1)
    Use -v to see a list of the files searched for.

Utility/UserInfo.hs:16:8:
    Could not find module `System.PosixCompat'
    Perhaps you meant
      System.Posix.Temp (from unix-2.7.0.1)
      System.Posix.Types (from base)
    Use -v to see a list of the files searched for.

Did cabal install --only-dependencies and had to add --force-reinstalls . Which already looks like a bad idea.

Failed with

Configuring gnuidn-0.2.1...
setup-Simple-Cabal-1.22.0.0-x86_64-linux-ghc-7.8.4: The program 'c2hs' is
required but it could not be found.
Building hinotify-0.3.7...
Configuring gsasl-0.3.5...
Installed edit-distance-0.2.1.2
Configuring regex-posix-0.95.2...
Failed to install gsasl-0.3.5
Build log ( /root/.cabal/logs/gsasl-0.3.5.log ):
Configuring gsasl-0.3.5...
setup-Simple-Cabal-1.22.0.0-x86_64-linux-ghc-7.8.4: The program 'pkg-config'
version >=0.9.0 is required but it could not be found.

Installed pkg-config and c2hs and it passed further

Failed

cabal: Error: some packages failed to install:
dbus-0.10.9.2 depends on libxml-sax-0.7.5 which failed to install.
dns-1.2.3 failed during the building phase. The exception was:
ExitFailure 1
fdo-notify-0.3.1 depends on libxml-sax-0.7.5 which failed to install.
gnuidn-0.2.1 failed during the configure step. The exception was:
ExitFailure 1
gnutls-0.1.5 failed during the configure step. The exception was:
ExitFailure 1
gsasl-0.3.5 failed during the configure step. The exception was:
ExitFailure 1
libxml-sax-0.7.5 failed during the configure step. The exception was:
ExitFailure 1
network-protocol-xmpp-0.4.6 depends on libxml-sax-0.7.5 which failed to
install.

Installed libxml2-dev

Failed

cabal: Error: some packages failed to install:
dns-1.2.3 failed during the building phase. The exception was:
ExitFailure 1
gnuidn-0.2.1 failed during the configure step. The exception was:
ExitFailure 1
gnutls-0.1.5 failed during the configure step. The exception was:
ExitFailure 1
gsasl-0.3.5 failed during the configure step. The exception was:
ExitFailure 1

Installed libgsasl7-dev and libgnutls-dev

One remaining issue is

Building dns-1.2.3...
Preprocessing library dns-1.2.3...

Network/DNS/Decode.hs:16:8:
    Could not find module `Data.Conduit.Network'
    Use -v to see a list of the files searched for.

Network/DNS/StateBinary.hs:17:8:
    Could not find module `Data.Conduit.Attoparsec'
    Use -v to see a list of the files searched for.
cabal: Error: some packages failed to install:
dns-1.2.3 failed during the building phase. The exception was:
ExitFailure 1

So if i do cabal install dns-1.2.3 --force-reinstalls it configures the older versions of conduit-extra and that again breaks the cabal install in git-annex. Forcing reinstall in git-annex breaks dns-1.2.3 and it is one cyclic mess.

ghc-pkg check gives:

The following packages are broken, either because they have a problem
listed above, or because they depend on a broken package.
dns-1.2.3
conduit-extra-1.1.6.2
HTTP-4000.2.19

cabal install HTTP-4000.2.19 fixed the warning for http pacakge.

cabal install conduit-extra-1.1.6.2
Resolving dependencies...
In order, the following would be installed:
nats-1 (reinstall) changes: hashable-1.2.2.0 -> 1.2.3.1
semigroups-0.16.1 (reinstall) changes: hashable-1.2.2.0 -> 1.2.3.1,
text-1.1.0.0 -> 1.2.0.3, unordered-containers-0.2.4.0 -> 0.2.5.1
void-0.7 (reinstall) changes: hashable-1.2.2.0 -> 1.2.3.1
conduit-1.2.3.1 (reinstall)
conduit-extra-1.1.6.2 (reinstall) changes: conduit-1.2.3.1 added
cabal: The following packages are likely to be broken by the reinstalls:
contravariant-1.2.0.1
semigroupoids-4.3
reducers-3.10.3.1
tasty-rerun-1.1.3
profunctors-4.3.2
free-4.10.0.1
keys-3.10.1
kan-extensions-4.2.1
pointed-4.2
adjunctions-4.2
bifunctors-4.2
comonad-4.2.2
http-conduit-2.1.5
attoparsec-conduit-1.1.0
Use --force-reinstalls if you want to install anyway.

Trying to rebuild the environment with

cabal sandbox init
cabal install

in git-annex directory.

Still the same

cabal: Error: some packages failed to install:
dns-1.2.3 failed during the building phase. The exception was:
ExitFailure 1
git-annex-5.20150205 depends on dns-1.2.3 which failed to install.

@marin have you tried forgetting about getting the git-annex source code, and doing cabal install git-annex --prefix=/opt/gitlab/embedded?

@jacobvosmaer Thanks for the suggestion. Yes, I am there now. Up until my few last comments all the issues were about building the environment that would allow me to do that.

I think I am getting the "Cabal hell" with the dependencies so just using cabal to install git-annex is what I am currently trying out.

it takes some time tho...

Same thing with cabal install git-annex --prefix=/opt/gitlab/embedded

cabal: Error: some packages failed to install:
dns-1.2.3 failed during the building phase. The exception was:
ExitFailure 1

From a sandbox:

cabal install git-annex --prefix=/tmp -f"-assistant -webapp -webdav -pairing -xmpp -dns"
<snip>
[307 of 341] Compiling Command.AddUrl   ( Command/AddUrl.hs, dist/dist-sandbox-e946ccd1/build/git-annex/git-annex-tmp/Command/AddUrl.o )
[308 of 341] Compiling Command.ImportFeed ( Command/ImportFeed.hs, dist/dist-sandbox-e946ccd1/build/git-annex/git-annex-tmp/Command/ImportFeed.o )

Command/ImportFeed.hs:279:28: Not in scope: ‘getItemSummary’

Command/ImportFeed.hs:280:32:
    Not in scope: ‘getItemDescription’
    Perhaps you meant ‘getFeedDescription’ (imported from Text.Feed.Query)
cabal: Error: some packages failed to install:
git-annex-5.20150205 failed during the building phase. The exception was:
ExitFailure 1

Different failure.

Also, have you tried https://downloads.kitenet.net/git-annex/linux/current/ ?

@jacobvosmaer I have tried the standalone, tried building the package with it and the health check failed with 10s of warnings. I am also a bit scared to use that as it has its own git which would make maintenance even harder (having different versions of git).

I am going to try to build an older version of git-annex, the current one is very new: 20150205

cabal install git-annex-5.20141125 --prefix=/opt/gitlab/embedded fails with other package dependencies broken.

Yes it looks like the 'linux-standalone' git-annex tarball is a lot like an omnibus package.

Final steps:

--> /opt/gitlab/embedded/bin/git-annex

                      [HealthCheck] E | The following libraries cannot be guaranteed to be on target systems:
    --> /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc778b9e000)
    --> /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fc778842000)
    --> /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fc777fa9000)

                      [HealthCheck] E | The precise failures were:
    --> /opt/gitlab/embedded/bin/git-annex
    DEPENDS ON: libz.so.1
      COUNT: 1
      PROVIDED BY: /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc778b9e000)
      FAILED BECAUSE: Unsafe dependency
    --> /opt/gitlab/embedded/bin/git-annex
    DEPENDS ON: libxml2.so.2
      COUNT: 1
      PROVIDED BY: /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fc778842000)
      FAILED BECAUSE: Unsafe dependency
    --> /opt/gitlab/embedded/bin/git-annex
    DEPENDS ON: libgmp.so.10
      COUNT: 1
      PROVIDED BY: /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fc777fa9000)
      FAILED BECAUSE: Unsafe dependency

To get the haskell environment ready I did the following:

sudo apt-get install ghc pkg-config libxml2-dev libgsasl7-dev c2hs libgnutls-dev libncurses5-dev llvm m4

wget https://www.haskell.org/ghc/dist/7.8.4/ghc-7.8.4-src.tar.bz2
tar xf ghc-7.8.4-src.tar.bz2
cd ghc-7.8.4
./configure --prefix=/usr/local
make
make install

 wget https://www.haskell.org/cabal/release/cabal-1.22.0.0/Cabal-1.22.0.0.tar.gz
 tar Cabal-1.22.0.0.tar.gz
 ghc -threaded --make Setup
 ./Setup configure --user --prefix=/usr/local
 ./Setup build

wget https://www.haskell.org/cabal/release/cabal-install-1.22.0.0/cabal-install-1.22.0.0.tar.gz
tar xf cabal-install-1.22.0.0.tar.gz
ghc-pkg check # probably not needed, I had to do some checking at some point
./bootstrap.sh --global

wget https://downloads.haskell.org/~platform/2014.2.0.0/haskell-platform-2014.2.0.0-srcdist.tar.gz
tar xf haskell-platform-2014.2.0.0-srcdist.tar.gz
vim hptool/src/OS/Win.hs comment out line 14
./platform.sh

For CentOS I had to start off with:

yum install ghc
yum install llvm
yum install ncurses-devel zlib-devel
yum install gnutls-devel libxml2-devel libgsasl-devel gtk2hs-buildtools
# at some point i had a failure with c2hs but cabal was already installed so I could do:
cabal install c2hs

We're not sure which Haskell compiler version we need.
We now have trouble installing a modern Haskell compiler on our build box.
We need control of which libraries are linked against the executable, it keeps grabbing libraries that are not on the target server.

We're considering shipping it without compiling, people will have to do apt-get install git-annex

For 7.8 we are shipping without compiling.

Milestone changed to 7.9

I think there are three parts to this puzzle.

install ghc + cabal on the build machines
compile git-annex during the build
make sure that git-annex only links against libraries in /opt/gitlab

The middle problem is solved:

mkdir sandbox && cd sandbox
cabal sandbox init
cabal install git-annex -f"-assistant -webapp -webdav -pairing -xmpp -dns -s3 -inotify -dbus -testsuite -feed -quvi -tahoe -desktopnotify -torrentparser" --prefix=#{install_dir}

About ghc + cabal. cabal is a command line tool that acts like Ruby's bundle; the cabal command line tool is distributed in a package called 'cabal-install'. Internally, cabal the CLI tool relies on a library called Cabal. When you install ghc (the Haskell compiler) you get a version of Cabal (ghc 7.8.4 has Cabal 1.18.1.5), so really all you need is to install on the build boxes is ghc and cabal-install.

To compile ghc from source, you need a working ghc compiler on your system. (Yay, bootstrapping.)

On EL 6, yum install ghc gives you a version of ghc that is too old to build ghc 7.8.4 with. :(

Luckily you can also get binary tarballs for ghc. So in theory, all we need to do to install ghc on the build box is install from a binary tarball.

I mentioned above that in theory, all you need is ghc + cabal-install. But that seems to be unreliable. What we can also do, and what seems more reliable (on some platforms), is to install a newer version of Cabal, and then a matching newer version of cabal-install. I got this working on Debian 7, but not on EL6 or EL7.

install ghc 7.8.4 (binary package)
install Cabal 1.22.0.0
install cabal-install 1.22.0.0

This problem is open at the moment.

Finally, if you do get a ghc 7.8.4 + cabal going on Debian 7, you get the problem that git-annex links against the wrong libraries.

jacobvosmaer@debian:~/sand$ ldd ../bin/git-annex
        linux-vdso.so.1 =>  (0x00007fffd6dff000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f73ced02000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f73ceafa000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f73ce8f7000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f73ce6f2000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f73ce483000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f73ce201000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f73cdfe4000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f73cddce000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f73cda42000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f73cef26000)

We want to use a different libgmp, and a different libz. (I believe the other libs are white-listed by Omnibus.)

To get linked against other libraries we will need to use ghc (compiler) flags. We don't speak directly to ghc, so this will probably mean either tacking flags onto cabal install git-annex, or editing .cabal files, so cabal tells ghc the right thing.

In short the problems are:

Install ghc+cabal on our build boxes and control which libraries git-annex links to.

cc @sytses ^^^ less than 14 characters

What I have so far for the 'install ghc+cabal' part is in gitlab-omnibus-builder!1 (closed).

@jacobvosmaer :)

sudo apt-get instal git-annex installs the following packages:

 git git-man git-remote-gcrypt liberror-perl libgmp10 libgsasl7 libice6
 liblua5.2-0 libntlm0 libquvi-scripts libquvi7 libsm6 libxt6 nocache quvi
 ssh-askpass x11-common

Milestone changed to 7.10

Milestone changed to 7.11

Back to this.

After getting to the point where I was 2 months ago, I've decided to go in a bit of different direction. I already know which libs are an issue so I am currently trying to compile alex or happy package which are the ones that are linking against libgmp that is outside of the whitelisted path.

I am using cabal to install them, with the extra flags --extra-lib-dirs=/opt/gitlab/embedded/lib/ --extra-include-dirs=/opt/gitlab/embedded/include/ where libgmp is.

After several attempts and tweaks the linking stays the same:

linux-vdso.so.1 =>  (0x00007fff91dff000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f2f863a9000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2f861a5000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2f85f87000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f2f85d19000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2f85a1d000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2f8565d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2f865b9000)

I've found a report that ignoring extra-lib-dirs is possibly a bug, with the label bug set 10 hours ago :/

Milestone changed to 7.12

Still no progress on this, tried compiling GHC with a compiled version of libgmp on a non standard location and git-annex somewhere still builds against the OS built version.

poor you @marin

I can understand if you want to put this off for a while. LFS has been the top request for quite a while now http://feedback.gitlab.com/forums/176466-general/suggestions/7502608-git-large-file-storage-lfs-support

@marin how hard is it to implement LFS? if we do that we can skip shipping Annex.

@sytses I haven't had any time to invest in investigating this at the moment due to more urgent customer issues.

@marin And that is fine, I wonder if we should invest time in this at all.

Milestone changed to 7.13

Milestone changed to 7.14

We made some move on LFS and so far it seems that users are content with installing git-annex using their package repo. Considering how much work this was and that it is not closer to being done I will close the issue until the need is back again.

Status changed to closed

+1, this was such a headache

Add git-annex library to the package

Designs

Child items ...

Activity

Admin message

Admin message

Add git-annex library to the package

Activity