GNU Wget

Getting started with GNU Wget2.

- Aniketh Girish

2018-04-28





Hello everyone,

I can ascertain you that, you don't need a specific blog post like this to get a grab on how to start contributing to GNU Wget or Wget2. But then, there are a few things that would help you to understand quite easily along with this blog since I would likely be describing issues I faced. Also, being an active member of [email protected], this blog post would likely be served as a good material for the juniors to come for a easy start on the project.

Before we move on, I can hear your mind mumbling, "oh there is Wget2 as well, why on earth do we need Wget and Wget2? :P

Well, here it is: GNU Wget2 is the successor of GNU Wget, a file and recursive website downloader. It is build wrapping around libwget, which is the core library for Wget as well. Further, Wget2 handles multiple features that don't exist in Wget such as, Support for HTTP/1.1 and HTTP/2.0 protocol, TLS Session Resumption including persistent session data cache, TLS False Start,  HTTP2 support via nghttp2 and GnuTLS ALPN including streaming/pipelining, support for HTTP Strict Transport Security, DNS lookup cache,IPv4 and IPv6 support and much more. Wget2 works multi-threaded and uses many features to allow fast operation.

So, let's get started. This will be solely on Wget2 even though Wget also follows the same procedure.

Start cloning the project from the GitLab mirror.

~$ git clone https://gitlab.com/gnuwget/wget2.git
~$ cd wget2
~$ ./bootstrap

The following packages are needed to build the software

  • autotools (autoconf, autogen, automake, autopoint, libtool)
  • python (recommended for faster bootstrap)
  • rsync
  • tar
  • makeinfo (part of texinfo)
  • pkg-config >= 0.28 (recommended)
  • doxygen (for creating the documentation)
  • pandoc (for creating the wget2 man page)
  • gettext >= 0.18.2
  • libiconv (needed for IRI and IDN support)
  • libz >= 1.2.3 (the distribution may call the package zlib*, eg. zlib1g on Debian)
  • liblzma >= 5.1.1alpha (optional, if you want HTTP lzma decompression)
  • libbz2 >= 1.0.6 (optional, if you want HTTP bzip2 decompression)
  • libbrotlidec >= 1.0.0 (optional, if you want HTTP brotli decompression)
  • libgnutls (3.3, 3.5 or 3.6)
  • libidn2 >= 0.9 + libunistring >= 0.9.3 (libidn >= 1.25 if you don't have libidn2)
  • flex >= 2.5.35
  • libpsl >= 0.5.0
  • libnghttp2 >= 1.3.0 (optional, if you want HTTP/2 support)
  • libmicrohttpd >= 0.9.51 (optional, if you want to run the test suite)
  • lzip (optional, if you want to build distribution tarballs)
  • lcov (optional, for coverage reports)
  • libgpgme >= 0.4.2 (optional, for automatic signature verification)

The versions are recommended, but older versions may also work.

Now let's start building our development environment.

Run,

~$ ./configure

After all the configuration is done, we should start compiling the whole project to produce the plugins and as to create the executable to install.

~$ make

After this install Wget2 and libwget.

~$ sudo make install (or su -c "make install")

Henceforth, the installation is finished. Simple right  :)

Now to the actual part of development.

  1. Reproducing the bugs that you are going to fix if any, else, making the appropriate changes to the code.
  2. Things move on as we test the application in multiple ways. So that it won't break time to time or neither during when it is running on different systems. After all, Wget is in 10 million systems already :)

Since we are using GitLab making a merge request isn't quite same as a pull request or a merge request in GitHub.

The steps are as follow:

  1. Create a branch that with a proper name of the issue origin pulled from upstream.
  2. Make the appropriate changes that you wish to make.
  3. Compile it, test it and see if you see the output that you wish to achieve.
  4. Then, make a Merge request to the upstream branch through GitLab.

One of the merits of using a dedicated branch itself for your fix is that every commit you make to this branch is automatically reflected in your Merge request as well.

So, now we need to run the tests in our executable and the main tools we make use of is make check and valgrind.

Certain test suites won't undergo if you haven't activated few packages enlisted in the requirements. Henceforth, give through reading through the required packages list. For example, the suite/test doesn't work unless you install microhttpd and rest of the wget2 will automatically detect the existence of microhttpd automatically.

Further, to test the code.

First, we have to test the code locally in our system.

~$ make check

Then test the code locally using Valgrind for memory mismanagement.

~$ make check-valgrind

If you want Valgrind memcheck by default:

~$ ./configure --enable-valgrind-tests make check

Moreover, if you wish to run valgrind memcheck individually on tests

cd tests VALGRIND\_TESTS=1 
./test-k

This is it, actually, work till you pass all the tests and voila! :P

That's it for now. Will catch you later ;)

Cheers.

Reference

[0] - https://gitlab.com/gnuwget/wget2/