X-Git-Url: http://www.git.cypherpunks.ru/?a=blobdiff_plain;f=gocheese.texi;h=ee620dc9330c0be0d8ba34c93fca51fd0ae349c5;hb=3020e814c1e4e23baf281f4ff7a1121dd54f8fe3;hp=8ad7b69db949fd0919d28e95cc2b8941c1fb5507;hpb=c576240f769376cce57823e24158ab4bdaae324c;p=gocheese.git diff --git a/gocheese.texi b/gocheese.texi index 8ad7b69..ee620dc 100644 --- a/gocheese.texi +++ b/gocheese.texi @@ -2,6 +2,10 @@ @documentencoding UTF-8 @settitle GoCheese +@copying +Copyright @copyright{} 2019-2020 @email{stargrave@@stargrave.org, Sergey Matveev} +@end copying + @node Top @top @@ -10,11 +14,25 @@ GoCheese is Python private package repository and caching proxy. It serves two purposes: @itemize -@item hosting of private locally uploaded packages (conforming to - @url{https://www.python.org/dev/peps/pep-0503/, PEP-0503} (Simple - Repository API)) @item proxying and caching of missing packages from upstream - @url{https://pypi.org/, PyPI} + @url{https://pypi.org/, PyPI}, conforming to + @url{https://www.python.org/dev/peps/pep-0503/, PEP-0503} + (Simple Repository API) +@item hosting of private locally uploaded packages, conforming to + @url{https://warehouse.pypa.io/api-reference/legacy/, Warehouse Legacy API} +@end itemize + +Why could you like it and how it can be better to fit your needs? + +@itemize +@item No database required. Only filesystem storage with few simple + files per package. Package deletion, renaming, making it uploadable + (private) is done with simple @command{mkdir}, @command{touch}, etc + commands +@item Just single statically compiled Go binary +@item No configuration file, but several simple command line arguments +@item Consistency (because of atomic synced operations) and integrity + (because of SHA256 checksums stored nearby) @end itemize Initially it was created as a fork of @@ -22,47 +40,84 @@ Initially it was created as a fork of but nearly all the code was rewritten. It has huge differences: @itemize -@item proxying and caching of missing packages -@item atomic packages store on filesystem -@item SHA256-checksummed packages (both uploaded and proxied one) -@item no TLS support -@item no YAML configuration, just command-line arguments -@item no package overwriting ability +@item Proxying and caching of missing packages, including GPG signatures +@item @url{https://pythonwheels.com/, Wheel} uploading support +@item Integrity check of proxied packages: MD5, SHA256, SHA512, BLAKE2b-256 +@item SHA256 checksums for stored packages +@item Verifying of SHA256 checksum for uploaded packages +@item Storing of uploaded GPG signatures +@item Secure Argon2i (or SHA256) stored passwords hashing +@item No YAML configuration, just command-line arguments +@item No package overwriting ability (as PyPI does too) +@item Graceful HTTP-server shutdown +@item Atomic packages store on filesystem @end itemize +Also it contains @file{contrib/pyshop2packages.sh} migration script for +converting @url{https://pypi.org/project/pyshop/, Pyshop} database into +GoCheese one, including private packages. + GoCheese is free software, licenced under -@url{https://www.gnu.org/licenses/gpl-3.0.html, GNU GPLv3} conditions: +@url{https://www.gnu.org/licenses/gpl-3.0.html, GNU GPLv3}: see the file COPYING for copying conditions. +Please send questions, bug reports and patches to @url{gocheese@@cypherpunks.ru}. + +@insertcopying + @menu +* Install:: * Usage:: * Password authentication: Passwords. +* TLS support: TLS. * Storage format: Storage. @end menu +@include install.texi + @node Usage @unnumbered Usage To use it for download purposes, just configure your @file{pip.conf}: -@verbatim +@example [install] index-url = http://gocheese.host:8080/simple/ -@end verbatim +@end example -@option{-refresh} URL behaves the same way as @option{-simple} one, but -is always refreshes package versions from PyPI when listing it. You can -use it to forcefully update known package versions. +@option{-refresh} URL (@code{/simple/} by default) automatically +refreshes metainformation (available versions and their checksums) +from the upstream, when queried for package directory listing. +@option{-norefresh} prevents upstream queries. -You can upload packages to it with -@url{https://pypi.org/project/twine/, twine}: +@option{-gpgupdate} is useful mainly for migrated for Pyshop migrated +repositories. It forces GPG signature files downloading for all existing +package files. -@verbatim +You can upload packages to it with @url{https://pypi.org/project/twine/, twine}: + +@example twine upload --repository-url http://gocheese.host:8080/simple/ \ --username spam \ - --passwd foo dist/tarball.tar.gz -@end verbatim + --password foo dist/tarball.tar.gz +@end example + +Or you can store it permanently in @file{.pypirc}: + +@example +[pypi] +repository: https://gocheese.host/simple/ +username: spam +password: foo +@end example + +If @command{twine} sends SHA256 checksum in the request, then uploaded +file is checked against it. + +Pay attention that you have to manually create corresponding private +package directory! You are not allowed to upload anything explicitly +flagged as internal package. @node Passwords @unnumbered Password authentication @@ -71,9 +126,11 @@ Password authentication is required for packages uploading. You have to store your authentication data in @option{-passwd} file in following format: -@verbatim +@example username:hashed-password -@end verbatim +@end example + +Empty lines and having @verb{|#|} at the beginning are skipped. Supported hashing algorithms are: @@ -82,8 +139,8 @@ Supported hashing algorithms are: @item @url{https://www.argon2i.com/, Argon2i} (recommended one!) To get Argon2i hashed-password you can use any of following tools: @itemize - @item @url{https://github.com/balakhonova/argon2i, - go get github.com/balakhonova/argon2i} (Go) + @item go get @url{https://github.com/balakhonova/argon2i, + github.com/balakhonova/argon2i} (Go) @item @url{https://github.com/p-h-c/phc-winner-argon2} (C) @end itemize Example user @code{foo} with password @code{bar} can have the @@ -96,13 +153,14 @@ foo:$argon2i$v=19$m=32768,t=3,p=4$OGU5MTM3YjVlYzQwZjhkZA$rVn53v6Ckpf7WH0676ZQLr9 @item SHA256 You can use your operating system tools: -@verbatim +@example # BSD-based systems: $ echo -n "password" | sha256 # GNU/Linux-based systems $ echo -n "password" | sha256sum -@end verbatim +@end example + Example user @code{foo} with password @code{bar} will have the following password file entry: @@ -114,14 +172,36 @@ foo:$sha256$fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9 You can refresh passwords by sending @code{SIGHUP} signal to the working daemon: -@verbatim +@example $ pkill -HUP gocheese $ kill -HUP `pidof gocheese` -@end verbatim +@end example Before refreshing it's recommended to check @option{-passwd} file with @option{-passwd-check} option to prevent daemon failure. +@node TLS +@unnumbered TLS support + +You can enable TLS support by specifying PEM-encoded X.509 certificate +and private key files. Go's TLS implementation supports TLS 1.3, HTTP/2 +negotiation, Keep-Alives, modern ciphersuites and ECC. + +For example generate some self-signed certificate using GnuTLS toolset: + +@example +$ certtool --generate-privkey --ecc --outfile prv.pem +$ cert_template=`mktemp` +$ echo cn=gocheese.host > $cert_template +$ certtool \ + --generate-self-signed \ + --load-privkey=prv.pem \ + --template $cert_template \ + --outfile=cert.pem +$ rm $cert_template +$ gocheese -tls-cert cert.pem -tls-key prv.pem [...] +@end example + @node Storage @unnumbered Storage format @@ -130,28 +210,41 @@ Root directory has the following hierarchy: @verbatim root +-- public-package - | +- public-package-0.1.tar.gz.sha256 + | +- public-package-0.1.tar.gz.md5 + | +- public-package-0.1.tar.gz.blake2_256 + | +- public-package-0.1.1.tar.gz.blake2_256 | +- public-package-0.2.tar.gz + | +- public-package-0.2.tar.gz.asc | +- public-package-0.2.tar.gz.sha256 +-- private-package - | +- .private + | +- .internal | +- private-package-0.1.tar.gz + | +- private-package-0.1.tar.gz.asc | +- private-package-0.1.tar.gz.sha256 |... @end verbatim -Each directory is a package name. When you try to list non existent -directory contents (you are downloading package you have not seen -before), then GoCheese will download information about package's -versions with checksums and write them in corresponding @file{.sha256} -files. However no package package tarball is downloaded. +Each directory is a normalized package name. When you try to list non +existent directory contents (you are downloading package you have not +seen before), then GoCheese will download information about package's +versions with checksums and write them in corresponding +@file{.sha256}, @file{.blake2_256}, @file{.sha512}, @file{.md5} files. +However no package package tarball is downloaded. When you request for particular package version, then its tarball is -downloaded and verified against the checksum. For example in the root -directory above we have downloaded only @file{public-package-0.2}. - -Private packages contain @file{.private} file, indicating that it must -not be asked in PyPI if required version is missing. You have to create -it manually. +downloaded and verified against the stored checksum. But SHA256 is +forced to be stored and used later. + +For example @file{public-package} has @code{0.1} version, downloaded a +long time ago with MD5 checksum. @code{0.1.1} version is downloaded more +recently with BLAKE2b-256 checksum, also storing that checksum for +@code{0.1}. @code{0.2} version is downloaded tarball, having forced +SHA256 recalculated checksum. Also upstream has corresponding +@file{.asc} signature file. + +@file{private-package} is private package, because it contains +@file{.internal} file. It can be uploaded and queries to it are not +proxied to upstream PyPI. You have to create it manually. If you upload +GPG signature, then it will be also stored. @bye