Hi all,

I have graciously given up on my wish of regularly blogging my life or even my work :(. Hence another late blog. I had spent 2 months in Tokyo interning as a research associate in Internet Initiative Japan - Innovation Institute(IIJ-II) where I worked on Docker port to macOS - and hence such a post. As always, I will try to write another blog on my experience living in Tokyo for two months in another post - but let this post be complete technical.

I’m splitting the contents of this post into two. This post will contain the background study on what exactly are we looking at the port of dockerd for macOS and why we actually need that? Next post will be all about the work I did and the results.

Let’s start with understanding the background of docker and how docker works.

Docker is composed of two main components, a command-line application for users and a daemon which manages containers. The daemon relies on two sub-components to perform its job, storage on the host file system for image and container data; and the LXC interface(runC) to abstract away the raw kernel calls needed to construct a Linux container.

All these functionalities are tightly coupled with Linux kernel features and our main motive is to bring agility for the docker project to co-exist in multiple platforms.

Core Kernel dependencies were:

• Kernel namespaces (ipc, uts, mount, pid, network and user)
• Apparmor and SELinux profiles
• Seccomp policies
• Chroots (using pivot_root)
• Kernel capabilities
• and CGroups (control groups)

To port dockerd server side daemon of docker to macOS. macOS currently uses a hypervisor to run the dockerd daemon on linux kernel system. Our goal is to eliminate this middleware, let the dockerd daemon communicate directly with the darwin kernel instead of running the daemon inside a isolated hypervisor providing Linux kernel architecture.

The current scenario for macOS to run docker in the host system is that, it availes a docker client which runs in the host system itself while the server side dockerd daemon runs in the Linux VM/hypervisor. Therefore, for the client to connect with the server dockerd daemon it has to tunnel through this Linux hypervisor.

Our intention is to port dockerd daemon to run in macOS without an intermediate Linux hypervisor to let docker communicate with the dockerd server directly without an overhead of another stack running beneath. Which implies. We provide Darwin based support for docker to run dockerd independent of LKL based VM and henceforth, let the dockerd after the port to be able to instantiate a Linux application.

Docker engine exposes the rest API which can be used to control the daemon. The client, docker uses this API to give instructions to the daemon. The Docker client interfacing through dockerd CLI to connect with the dockerd daemon, listens for Docker API requests and this dockerd tracks everything related to Docker, including containers, images, volumes, service definition, and secrets.

Docker engine is responsible for running processes in isolated environments. For each process, it generates a new Linux container, allocates a new filesystem for it, allocates a network interface, sets an IP for it, sets NAT for it and then runs processes inside of it. It also manages such things as creating, removing images, fetching images from the registry of choice, creating, restarting, removing containers and many other things.

The above image illustrates the way how docker works on macOS. Our main goal is to eliminate the LinuxKit VM from this architecture. Docker daemon uses Linux kernel vastly to communicate with the host. We need to figure out where all Linux kernel modules, drivers are used and further port those to analogues darwin based code. Further, while we remove the LinuxVM kit, there will dependencies that are related to the hypervisor (virtualisation), networking and the filesystem part. We have to bring in all of that changes into the engine so that dockerd runs independent of Linux kernel.

We will start of with few minor ports which can be easily found out by compilation and building the dockerd engine in macOS system. This would give enough time to help us understand a bit of code architecture as well as would let us plan ahead on where are major changes has to be brought.

On Docker for Mac, clients can connect to the Docker Engine through a Unix socket: unix:///var/run/docker.sock. We have to debug and trace how this call goes and connect the dockerd daemon server and port most of the underlying code that the stack shows.

- Virtualization

As mentioned, currently this is how virtualization happens inside the docker for macOS:

- Networking

Our aim at this point is an effective way to reconstruct container traffic into separate TCP/IP flows and translate them into native OSX socket. Currently, it uses a VPN kit to channel the traffic between the client and the server.

We need to figure out a way to have DHCP, NTP as a standalone ethernet datagram as how docker for Linux illustrates brought into macOS as well by separating the needful module from the VPN kit. Henceforth, This lets the container to run and create socket with it to expose the container to localhost.

Next, There is no docker0 bridge on macOS Because of the way networking is implemented in Docker Desktop for Mac, you cannot see a docker0 interface on the host. That is, there is no docker0 interface on the host for macOS, we need to separate that from the hypervisor network management interface. This interface is actually within the virtual machine. We need to bring that out of the virtual system and integrate it directly to the engine.

- Filesystem sharing

Here in the filesystem segment, we have to figure out ways to communicate with osxfs system without a intermediate Linux host client like FUSE that provides API calls to osxfs system.

Namespaces

Docker uses a technology called namespaces to provide the isolated workspace called the container. When you run a container, Docker creates a set of namespaces for that container.

These namespaces provide a layer of isolation. Each aspect of a container runs in a separate namespace and its access is limited to that namespace.

Docker Engine uses namespaces such as the following on Linux:

• The pid namespace: Process isolation (PID: Process ID).
• The net namespace: Managing network interfaces (NET: Networking).
• The ipc namespace: Managing access to IPC resources (IPC: InterProcess Communication).
• The mnt namespace: Managing filesystem mount points (MNT: Mount).
• The uts namespace: Isolating kernel and version identifiers. (UTS: Unix Timesharing System).

Control groups

Docker Engine on Linux also relies on another technology called control groups (cgroups). A cgroup limits an application to a specific set of resources. Control groups allow Docker Engine to share available hardware resources to containers and optionally enforce limits and constraints. For example, you can limit the memory available to a specific container.

Union file systems

Union file systems, or UnionFS, are file systems that operate by creating layers, making them very lightweight and fast. Docker Engine uses UnionFS to provide the building blocks for containers. Docker Engine can use multiple UnionFS variants, including AUFS, btrfs, vfs, and DeviceMapper.

Container format

Docker Engine combines the namespaces, control groups, and UnionFS into a wrapper called a container format. The default container format is libcontainer.

This is the background study that we need to do a complete port of dockerd. Note, these changes are not just limited to dockerd alone but can span across containerd, runU and much more.

Due to the time constraints, I couldn’t explain each and everything on how docker works exactly, but you can find other blog posts on the same. This majorly is a proposal on how I think dockerd could be ported to macOS without a Linux VM enacting as middleware and hence directly communicate with the host and provide a general cross-platform docker implementation.

This post contains the rest of my work and how I achieved a cross-platform docker implementation:

https://anikethgirish.wordpress.com/2019/09/14/dockerd-port-for-macos-part-2/

Happy reading peeps!

Cheers.

Hey!!,

This is the continuation of the last post since I thought once I gave an introduction about what docker is and what this project was all about, it is better to have the rest written as a separate blog since it will be easier to skim through :D

The dockerd port to macOS eventually started as a subset research goal carried in the redesign of container stack to experiment and measure extensible and platform independence attained in containers on µKontainer of which a library operating system was introduced which decouples the kernel component and uses that as the container kernel within pure user space processes. On this light, docker has introduced a light guest VM for docker to run on macOS - while docker in Linux run directly on the host machine. With the introduction of µKontainer docker could be run directly on the macOS host. Therefore, this port of dockerd on macOS looks at the implementation of docker in macOS with the help of µKontainer and henceforth attain an implementation of docker independent of the host kernel it runs on.

With the context of the image above, docker for macOS heavily relies on the guest Linux VM to emulate missing components of Linux kernel in darwin host. Due to this advent of the guest Linux VM, Docker daemon is intertwined vastly with Linux kernel to communicate with the host. Henceforth, this makes it impossible to run dockerd directly on the host machine and rather has to use an intermediate guest VM - where the dockerd, the daemon side of docker exists within the VM while the docker client on host. Our proposed fix with µKontainer improves this scenario of the existence of dockerd on macOS host without this intermediate guest VM to run docker independent of the host.

Introduction of µKontainer does not only promises the platform independence attained by docker, but it also provides enough extensibility to introduce analogues components/features into the docker engine we introduce.

The low-level OCI Runtime environment, runU handles the creation of containers. Within the context of runU, it does a fork+exec call to initiate calls towards LKL (libOS) and the application to be executed. Containerd enacts as a process that could be used to manage these containers to build and start the execution of the application within the scope of these containers. Further, Docker engine/dockerd is responsible to run an application over the containers and scale.

Docker in macOS is split into two processes; docker client and dockerd daemon. Docker client uses dockerd daemon to communicate with the container processes.

With frankenlibc, and runu under ukontainer github-org, you can run Linux application on macOS. we’ve also integrated this with docker image, which can be used with containerd and ctr utilities.

The docker image which we created, named runu-base[1] - this image gets executed since the image contains cross linked binaries of Linux specific programs compiled and build from the respective program-specific source code.

on a Linux host, you can run docker run alpine uname -a with LKL-ed kernel (with Alpine Linux official image). This major focus of this project was to port dockerd daemon, which currently only supports Linux to macOS.

Currently, docker on macOS uses guest Linux VM using Hypervisor.framework exposed by darwin kernel(macOS) to run the dockerd daemon on darwin kernel system. This hypervisor framework helps the VM to emulate a physical environment for the Linux kernel to act as an intermediate on darwin. Our goal is to eliminate this middleware(guest VM), let the dockerd daemon communicate directly with the darwin kernel instead of running the daemon inside an isolated hypervisor providing Linux kernel architecture.

Initially on the port, learning more about the existing codebase, we realised that instead of porting each subset of features in the codebase as such, it was better to focus on the port which would let us completely execute docker container run.. command. Because, we learned that docker container run has dependency over docker create, docker start and much more internals of the code. This way we don’t need to look at just one aspect of the codebase but instead port major components at one go.

To achieve this subset of the research project - the port of docker engine was divided into mainly three aspects:

1. Operating system-level port: The codebase of docker engine doesn’t support the direct compilation of docker engine on darwin platform. One of the tasks over this project was to extend the codebase to avoid such build/compile constraints and hence have the codebase to be platform-independent.
2. Runtime level port: In the codebase, there exist components which should be executed only during runtime on specific platforms. This type of constraints blocked the docker engine port to successfully run over darwin as it neglected much-required features that should also be in darwin to build/run. Our work during the phase of runtime level port focused on exposing such aspect of the codebase to darwin as well. Apart from this, one of the major goals of this phase was the streamlined integration of the code in order to get docker engine running irrespective of the platform it runs on. Since there was a particular subset of components integrated that would only work in a specific platform, such runtime level segregation helped to improve the code integration. Henceforth, this led us to include platform-specific code, yet, this integration wouldn’t break the system as a whole either.
3. The system features port: Apart from exposing features like chroot, exposing mounting options and establishing ociSpec use for darwin the majority work for the port relied on the porting of Libnetwork, Libcontainerd, moby/buildkit modules.

Note: Currently, our port of docker engine is supported only over VFS via graph driver module in docker engine.

Libcontainerd package of docker engine is a collection of gRPC client API for containerd. It initiates a gRPC client to invoke containerd process Since this package mostly handled the gRPC POST requests from the client to create, start, restart etc on the containerd process. A notable use case of libcontainerd is that at first it tries to spawn a dockerd managed containerd process. If that fails, then the only dockerd tries to spawn containerd as a different process. Most of the port here was only to attain platform independency of libcontainerd package which was Linux only and to extend it for darwin as well.

The libnetwork project of docker handles the networking of containers. The current docker libnetwork port consists of streamlined integration of darwin platform-specific code and build constraints in order to let docker engine use libnetwork APIs into the docker engine in darwin.

The buildkit API dependency of docker engine had to go through a series of changes in order to let its snapshotter control the filesystem to build the bundles properly and mount irrespective of the platform it is running upon after deflating the images. Updating the codebase of buildkit, we saw there were Linux only specific code like seccomp, speccov etc which were silently ignored over runtime while on darwin or introduced missing components in order to let docker run properly irrespective of the platform.

Another aspect of our work to port dockerd for macOS heavily relied on silently overriding the namespace and cgroups Linux only features. Containers are implemented using Linux namespaces and cgroups. Namespaces let you virtualize system resources, like the file system or networking, for each container. Cgroups provide a way to limit the number of resources like CPU and memory that each container can use. At the lowest level, container runtimes are responsible for setting up these namespaces and cgroups for containers and then running commands inside those namespaces and cgroups. Low-level runtimes support using these operating system features.

Additionally, we added support of another dependency of sysconf [1] for Go, without using cgo or external binaries in libcontainer module in opencontainers/runc to get clock ticks. Which measures CPU ticks between the start and end of a process. This integration of sysconf would help the future of docker engine port to include system platform-independent system configuration related functions calls in APIs directly which used to come from host kernels.

After the port of dockerd on darwin, we successfully support docker container to do major functionalities like create, run, pull etc.

Work in progress

Containerd process spawns multiple Childs as containerd-shim according to the number of containers running. Top to down, containerd initiates the call to create the containers and through this containerd-shim process, gives runU the action to create the containers.

Once runU creates the container properly and containerd is able build and handle the container, the low-level runtime runU process exits from the process chain and the container is handled by containerd-shim process alone.

That is, containerd-shim allows the runtime, runU to exit after it starts the container: This way we can run daemon-less containers because we are not having to have the long-running runtime processes for containers.

Once the docker run is successfully executed, ideally, containerd-shim process should exit as well. In the current scenario, it doesn’t happen. One of the reasons for this is that the filesystem implementation (mount/unmount) is not well implemented to be used in darwin and henceforth docker engine is not able to find a rootfs path to currently mount and unmount. We have landed up in a few workarounds to do this, but that is not ideal.

Apart from the thehajime/runu-base:0.1 image we are not able to run another image, for example, an alpine image. Initially, the image as such wasn’t pulled/downloaded from the docker hub since it is unlikely to run Linux image over darwin host directly without a guest Linux VM. Currently, We have introduced a patch that would let this image to be successfully downloaded into the host system. Once the image is unpacked as bundles, docker engine fails to flatten these layers properly. We suspect this also occurs due to the inadequate implementation of graph driver in darwin.

Future additions proposed

Improvement of VFS to actually mount and unmount functionalities and the rest of the features that it offers. Along with this, maybe it should be possible to add other filesystem support as well.

Improvement on the libnetwork port. Current port of libnetwork only cares about the system platform independence while there still need to do a serious amount of work to actually use the libnetwork as a container networking platform. It would be ideal to replicate something similar to docker0 bridge which actually exists in Linux do communicate with the client and the server.

Code refactor: There exist stub functions which would be useful if we fill it in. As well as there exists a lot of hacky workaround in the codebase. To actually able to add our work into the upstream moby repository - it would be ideal to refactor the code in a better way.

Reference

1. https://docs.docker.com/engine/reference/commandline/dockerd/
2. https://docs.docker.com/engine/docker-overview/
3. http://collabnix.com/how-docker-for-mac-works-under-the-hood/
4. https://github.com/docker/docker.github.io/tree/master/docker-for-mac
5. https://www.slideshare.net/AnilMadhavapeddy/advanced-docker-developer-workflows-on-macos-x-and-windows?ref=https://blog.docker.com/2016/05/docker-unikernels-open-source/
6. https://ieeexplore.ieee.org/document/5541547/
7. https://hub.docker.com/r/thehajime/runu-base

Finally, it’s time! :) I have been struck with this idea of building a Twitter bot recently, which would be curating a list of all relevant contents regarding Networking/Internet protocols, network security and future internet architecture which would make my life easier. So, this post serves as my official release of the bot to the public and I would certainly wish if I could get some inputs to make it better as this bot is just a work in progress :)

Well wait, how precisely would this help to make my life easier? Well, let us rewind a bit.

Back-story short, I have been indulged into learning more about Network protocols and future internet architecture with an interest in its design and security aspect of it. Well, cutting myself off from most of the social media, I went directly through to twitter. Where I found that the number of contents, discussions, opportunities to learn regarding my area of interest is tremendously enormous. I used to scroll day and night through Twitter to get all those fascinating and exciting feeds about my area of interests. Right now, my timeline would be just filled with contents related to network protocols and the major discussions around it. But then, it started to become a bit overwhelming to keep a tab on all of those resources and I wished if I could just get most of those relevant contents at one place.

So, here you go; The idea of Nawab was born. I have just devised a simple bot that would just retweet all the relevant posts or discussions on Twitter into the bots profile timeline. It certainly needs work on and with my other commitments lately, I am not pretty much sure to put my entire effort into this product; so it would be just perfect if I get more reviews and your views on the bot and improvements that I could add upon this one. :D

Github: https://github.com/Team-SYNACKd/NaWaB

Cheers.

Recently, I along with Abhinand ettan, Anu Chechi, Devi Chechi and Vibhoothi flew all the way to The Netherlands to participate in a hackathon. This hackathon was called hackathon for good and organised by The Municipality of The Hague with the motto to built and find solutions for Peace, Justice and Security. It was our first international hackathon and we were all super excited to spend our 36 hours hacking into an awesome project that we poured our heart and soul into from weeks before the event. In this short blog post, I will mainly focus on to describe what our hack was on and what we hacked during the entire 36 hours. Here we go ;)

We all are worried about dangerous people walking down the street just because we cannot read the information fast enough. As part of the evidence gathering and analysis process, the International Criminal Court has to analyse a number of written documents which can be a huge data set containing various information. The analysis is completely manual. The International Criminal Court has to go through all these information manually, figure out various entities from each file and layout relationships between those entities. Because of this slow and manual process, criminals spend more time outside rather than doing their time inside their cells. As this is a major issue the world is facing, so International Criminal Court had hosted a Challenge in the Hackathon for Peace Justice and Security.

Henceforth, we thought to pick this up because we felt that we could actually help to solve this problem. Even this being a tedious task, the thought that our solution would highly benefit lives of people at International criminal court and to help citizens of our nation to walk around freely in their neighbourhood without being scared of such criminals was the best motivation we had.

We introduced a hack, which we named TINA(Text mIning aNalysis dAtabase) to address this problem, yes we know that sounds a bit weird, that is weird Acronym.

TINA aimed at helping the International Criminal Court, by providing a simple, yet conclusive and complete automated application which takes care of all the Witness statements, NGO Reports, Media Articles, and much more and produce the expected entities along with the relationships it possessed by the application of various Machine Learning, Artificial Intelligence and deep learning techniques which is served as a whole package under the name of TINA.

What International criminal court expected was just a node + edge graphical representation stored over graphical database format – such as a .json file or a .csv. But as a team, we thought this isn’t enough. This can provide the International criminal court to identify the entities and relationships among the information but it wouldn’t speed up the process as it should be because we are talking about a very large dataset with the very high number of nodes and edges representing a crime alone maybe.

Henceforth, we decided to make sure Tina represent the data and visualise them in ways that would not only help to identify entities or the relationships that exists but will list the entire details in a dashboard. We included multiple features into the dashboard such as Crime Heat Map, Google Maps based Location Searching, Smart Table, Graphical Representation of the Data Extracted.

With the aim to provide a complete automated package to the International criminal court, we broke down the entire process into different parts for the application:

• Evidence Analysis
• Extractor part
• The dashboard
• The Core (ML/NLP)

Evidence Analysis

This section could be considered as a formal training for ourselves to identify the core aspect of the project. As a part of Evidence analysis, we tried hard to gather as much as dataset get some valid data to train our models. The process to understand the dataset provided by the ICC was hard until an extend - mainly because we only saw very less usable data points that can be used to layout the model. Henceforth, we decided to spend a considerable amount of our time during the hackathon just to find out dataset that could at least be used to as an example to train. We faced some amount of difficulty to find various resources from where we will get such important documents, even if it be bogus. But we were successful up to an extent to collect enough dataset for our project to train for but we couldn’t expect much rate of accuracy from those datasets though.

The Extractor

Next, it was about how to collectively process this entire dataset that the International criminal court would have. Since it these documents was of different file formats, we had to process them. Further, International criminal court would have this as a dataset of 10s or 100s of GB, so it would be highly unlikely for them to process them one by one. So, built an API which can take all the files like PDF, docx, txt and jpg irrespective of its file type and makes it to text which could be passed on to the next module of TINA. The input for this API can also be some folder, it will provide the OTP more flexibility like never before. The folder can contain any sort of data which is needed to be extracted. We went a bit out of the scope for giving a better User Experience (UX.). The Extractor can also take inputs as a URLs or any sort of web links to the Office of the Prosecutor at International criminal court can have an easier way to feed information directly through the internet. In the future, we are planning to integrate a proper Google Search feature inside the dashboard so instead of the linking, a whole new option which will be easier. Then after the extraction, we will be writing this into different files and also in a single file which appends all results as TXT along with the file metadata so that we don’t lose any information regarding a crime that can be useful in all ways possible. After parsing, this text file is passed to the Core module of Tina where the ML and NLP are processed and the detailed things about that later in this post below. Below you may find a snapshot of how our Office of the Prosecutor at International criminal court could interact with the extractor API through our dashboard.

The Dashboard

The dashboard is the core aspect of our project, We have specially designed the Dashboard with only one thing in mind that everyone who is 20 or who is 70 can use at the same time without being complex. That is, we really wanted our tool to be easy to use and doesn’t require someone to read through huge documentations to have a basic understanding of how to use it. For Instance, a person who doesn’t have much knowledge on understanding the directed Graphs it will be nearly not possible for him to read from the graph so we made integration of Google Maps for showing the Crime Location instead of showing the Location as a label. We have lots of numbers in where X person Killed Y people so we made that as an input for visual Graph. Also, we have made a special Heat Map of Crime, where all global crime details are shown. Below, you can find a few screenshots of how the dataset is being represented in various forms through our dashboard.

The Core( ML/NLP)

The core part of the project mainly focuses on automating the process of analyzing the evidence of various crimes that needed to be handled by ICC. After The extractor API did its black magic, the text is moved into the ML module to extract entities and find relationships between them in the context of ICC easily. While coding, we made use of various python packages like Spacy, nltk, keras. numpy and sklearn to do the needful. In order to achieve the process of identifying entities (or nodes) such as events, dates, places etc, we have used an approach which can provide better results. spaCy is one of the powerful natural language processing tools which excels at large-scale information extraction tasks. Combining the power of spaCy with LSTM to identify the nodes was one of the novel methods which we have used in this tool. Moreover using the output from spaCy as an input to LSTM gave us better results in cherry picking entities from the huge amount of text data.

One of the main challenges was that we had very less ground truth or factual data that were given to us. We focused on designing a better model which would work with a very minimal amount of data and further produce better results. We used the classification technique in order to train the model for classifying the relationships. So when a new data is fed to the model it will classify the relationship among the entities. For eg: it will classify that event and date is connected ie the event happened on XYZ date.

Keeping future in mind, if we plan to improve the model even further by using character-level embeddings we could use used implementation provided by the keras-contrib package, that contains useful extensions to the official keras package where it could also help to have a large number of datasets with more complicated named entities to have a stronger and well-trained model.

The technical stack

As a techie, this blog post is only half done without TINA’s technical stack ;)

The above figure illustrates the entire workflow of TINA in one place. Providing a brief explanation on this: At first, we have the entire dataset, let it be files of any kind - PDF, DOC(X) etc, this can be all in just one folder. It is then dumped into the Extractor API. All the black magic to extract the texts from those various files will be taken care of by the Extractor API. Further, the results from the API to the NER and REL module. Where NER module makes use of classical Natural Language Processing methodologies to extract various entities from the texts. Further, the REL Library makes sure the extracted entities have a better contextual meaning and finds a relationship with the entities extracted. Algorithms like LSTM works behind this library to help us to do that. Finally, we draw all our inferences through either via Geo Maps, graphs, CSV or heatmap through the dashboard that we designed using by Angular 6, Typescript as front-end stack and Django python web framework to support the backend.

You can find the Codebase here: https://gitlab.com/Aniketh01/tina

Cheers.

The last few months were something special - something different. It was an emotional rollercoaster, brought me down, tore me apart. In spite of them, there was the experience of my lifetime, best moments of my life ever and this post is all about it. I was lucky enough to get an opportunity to spend a month in an international university and thereby I spend almost 1 month as an International summer exchange student at Ben Gurion University, Israel.

“Decide whether or not the goal is worth the risks involved. If it is, stop worrying.”

In a nutshell, Each second I spend added to my life learning. It was rigorous - exciting - scary and just a perfect medium to introspect the actual me and who I am and how I react to things out of my comfort zone.

By inspecting the life of foreigners and moreover we as a third person in an entirely different situation, I think I reached to an understanding and could be said as the best takeaways for me from the entire trip would be: Life is short. Enjoy the most out of it. Never left out something or procrastinate because we cannot assure yourself about what will happen tomorrow. Do what you need to do, say what you wish to say, right away. Because you should never regret it later. Because we always see the situation that we are in at the point of time while we are blindsided to see the bigger picture. The other best learning I would say is that how to stand for yourself and stay put on your decisions. Something else that which keeps ringing inside my head is that never let others opinion and destroys your inner peace. Everyone might have heard this. Even I did too. But this was different. I experienced it. I believe it’s hardcoded into me.

I started off this fantastic adventurous journey at the start of July (1st of July) and I am writing this down when it’s time to wrap up the entire trip and while just hours before I boarding the return flight to India(IND). Well, you know, I got lazy and stopped it in the middle and right now I am completing this post in my hostel room at Amritapuri. Thanks to a friend of mine, Amrit Sreekumar who hosted me for the night before and dropped me off till the airport in Kochi. I think I should start from there as the initial/starting point of this entire journey. The flight journey was quite nice. Firstly good experience with Air India and a book in hand to help me strike out boredom.

I landed in Israel by 2nd of July midnight probably and went straight to the accommodation that was arranged for us by BGU. Rather than saying that they arranged an accommodation for us wouldn’t do justice for exactly what they did for us in co-operation with the Ben Gurion Campus tower. The welcoming was so sweet and enriching. Who does like to see a welcoming sweet note at your doorstep when you enter the place that you are going to live in for the rest of the month? :D

Let me start off by talking about the whole BGU and Beer Sheva experience. The course - most of it engaged with classes in Machine learning, Security and a great exposure towards leveraging Machine Learning to bring down Cyber Security threats that prevail in the society. This week mostly included roaming around the whole Beer Sheva, our Ben Gurion University and the place where we were accommodated. Beer Sheva is beautiful, BGU is just amazing - vibrant students culture and amazing university architecture. There was a building just left for students to conduct their programs as they wish for and engage in social activities. They screen different movies in different weeks and they have theatres inside the university itself for that. Moreover, during the FIFA football world cup fever, they screened the game in the student’s centre. And to enjoy the game we had unlimited beer, popcorn, coke and much more. (PS: You have to pay though :P). One thing that is evidently notable is that they love their country and they portray it everywhere. You could find Israel flags all around the place, in apartments - in houses - in cars etc. Further, The sunset over there is just love <3. You can find those beautiful pictures that I captured during the entire trip here and I have attached a few pictures of the sunset above.

During the first week at Israel, we were lucky enough to witness the socio-cultural fest that happened in the old city of Beer Sheva. Good amount of fantastic street food, Pub and clubs open all night, gathering arranged at a central location with music and dance - just the perfect way to spend your weekends :)

One thing that I loved about Israel was that everything that happened around there was well organised and the traffic policy that they followed was terrific. Pedestrians come first always and that is the best thing to be followed by a human being on the road. “Inside of us - it truly did make us feel that we own the street bruh :P”

This is where the fantastic people I met would come into the picture. Dana, Rotem, Amit, our professor Dr Nir Nissim and much. Few of the very best people whom I have met in my whole life. The enthusiasm with which they work and develop this upbringing of an international summer program is tremendous. The counsellors were a great support for every little detail we had. The people we meet. Other than that, my colleagues were great. Different - the mixture of people around the globe. China, India, Russia and Greece. Further talking about Israeli’s in general - most of them are really healthy and stays fit unlike in India :P. Maybe because their appetite is incredibly huge as compared. Next thing would be that - because they are this fit, most of them are quite big as compared to the physique Indian, especially south Indians carry. Talking about Indians, they love India and Indians a lot. They take vacations for like 2-6 months and travel all around India. Most of the Israeli’s whom I met and engaged in a conversation outside from BGU had visited India and talking to me had made me miss our country :P. Sweet thing to hear :D.

We roamed around in different places in Israel such as the Old City of Beer Sheva, Tel Aviv, Jaffa, Jerusalem, Ashdod and much more. Few as a part of the field trips will the others to just enjoy the place - get along with the whole community and experience the country. Few of my bucket list didn’t work out to be successful because of our tight schedule such as Haifa, scuba diving etc. I wish to go back soon and get that done as well ;). Another thing that I missed is to enjoy the nightlife of Tel Aviv - the party land. The night we choose to stay in Tel Aviv was the worst day anyone could have chosen because it was a public holiday :(. Even though a club was open late night, we were denied to enter :( - long story. (Long story short - age limit was of 24 :P). But still, we roamed around Tel Aviv the whole night until 3 AM. Ahh, the place we crashed for the night is called Roger hostels and it is just a great place to spend a night. We get to mingle with different people staying in your dorm and learn from the experience. Of all the places where we went to the Ashdod and Jerusalem was the best of all. It was just beautiful and lovely. You will never realise how fast time went away. The beaches are just beautiful, alluring, pleasing and what not. The turquoise water and the vibrant people around the place are just put me in an aww <3. The memories weived out there is incredibly a lot - writing down each line brings other memory into my head.

Ever heard of falafel and Hummus? That is the best and the authentic food that is served at Israel. It was good at first. But eating it continuously for weeks started to put an hatred in me for this. Almost all the time, my mouth craved for Indian food. It was probably the daily meal. Either falafel, schnitzel or the halo(aalo :P) paratha from the Indian restaurant I found out near the university called little India. At the end of 1 month - I just got sick of it. I was craving to have some good proper Indian food that was scarcely available and even if we found a place to eat Indian food it’s too costly for an Indian ;).

Well, other than the meal other things we’re sorted B). Well, in the morning, either cornflakes or bread with coffee. Then, at night, either I would cook rice (which is the only thing that I knew to cook properly and the resource that actually helped). Maggie noodles, soups, biscuits added up to this by the way ;). And this was just perfect. For a month though ;). I really started to appreciate our mess food which seems totally impossible to think of when you stay at your hostel. In the end, when I got so desperate to have a different dish, I even started to experiment with different varieties of things using rice. That made all the difference :P

One of the best thing that happened during the entire stay at Israel together with the whole summer program team was the potluck! It was just awesome. We had to cook some food and all of us in the program would share it with each other. Just had an amazing time with all of the colleagues in the program sharing food, memories and on top of it all - love! <3

Reaching the end of this post, the heartbreaking event would be the return back to India. I literally didn’t want to come back. I got accustomed to how Israeli’s lived and enjoyed their livelihood or moreover, I didn’t want to end my full of bursting surprising journey to end. I would miss the place - the entire country. The peeps out there. The whole OSP team. I never expected to create this whole of the best experience in my life ever with millions of memories made in just 1 month. When I landed in Israel, I saw a quote in the airport somewhere saying that ‘Once you visit Israel, you stay an Israel forever!’ well, me being an Indian right now, I’m a small Israeli at heart ;). Wrapping this post up - I wish to see Israel once again and hope to spend another fantastic summer there!

To an awesome life ahead!

Cheers.

The past few weeks have been challenging and exciting as well. I have never been involved in and engaged myself in something that has intrigued me this much. Could say that might be the interest brought in by computer networks or probably you could say the active and kind nature of my mentor ander juaristi.

Even though the wget community is small, but the people in the community is really free to help and let us learn even the simplest things. The first few weeks were a part of community bonding and I used this time to learn much more about my project DNS resolution over HTTPS and regarding multiple protocol suites to understand the better working condition of each. Further, I went through the codebase of wget2 too as to get a better understanding of the API architecture they follow throughout so that I could also write my API and code accordingly. Even the codebase is smaller as compared to the projects I have worked over the past few years. I don’t know probably this smaller codebase or the thought of me contributing could make a difference gives a feel of greater purpose with a satisfaction that I am actually learning something that would benefit me in the long term as well.

During the community bonding period rather than spending time on just reading the codebase and skimming through it and understanding other aspects as well, I thought of getting my hands dirty as well. Also, this will help my mentor and my community to see that I am actually working on getting things done and there is an urge to get things done.

I started working on a bug fix, which was a part of the statistical module that existed in Wget2, it was a simple fix. But then, few of the error that showed up because I lacked experience in coding with C and moreover, I lacked good quality coding habits as well, like managing memory, leaks that might happen and probably implementing a harder version of something rather than going to solve that in a simpler manner. Basically, I tend to reinvent the wheel rather using what exists.

Good things which happened while the struggle is that, I learned how to use GDB, how to use Valgrind at least in a small scale and hope to understand it in a deeper level as well. There were two mistakes that I remember now, ie, I used char* to store const char* and also I used xfree() to free the memory allocated by strdup() in the print function rather than in the free_stats() function which was already available. While in the learning process I have created notes as a Guide for the usage of Valgrind.

Almost near to the end of community bonding, I started to think and draw a sketch of how the resolver API should be like. At the beginning of this, my focus was diverted and my mind was totally on implementing a DNS library, again, reinventing the wheel. But then, Ander brought my focus back to the DNS-over-HTTPS resolver back again. He provided me with the skeleton of the API that we should be hacking around.

Few of the stub functions inside the API is which will be exposed throughout wget2 is:

void wget_dns_init(wget_dns_t *, wget_tcp_t *);

void wget_dns_deinit(wget_dns_t *);

void  wget_dns_set_config_int(wget_dns_t, int key, int value);

void wget_dns_set_config_string(wget_dns_t, int key, const char *value);

int wget_dns_resolve(wget_dns_t, const char *host, uint16_t port, struct addrinfo **out_addr);

Further, I have added the option –dns-over-https flag option to enable doh and let user tell wget2 that this type of resolver is active and added option –doh-resolver=[]. Which takes a string, which should be an IP address or a hostname of the server that supports DNS-over-HTTPS.

To get a basic understanding of how the DNS over HTTPS resolver actually works, I completed reading RFC’s for DNS and DNS-over-HTTPS. Moreover, I read through a few articles and other materials to understand how an actual resolver works and what is the difference here with our implementation and what exactly should be our end product. Further, I have been looking for many API’s available to produce the end product that we desire to make.

During the time, when I was reading the RFC’s to understand how things work, I was also trying to understand how the query should be parsed to HTTP/2 through TCP and how it is currently implemented in wget to find the possible options I have.

As a part of the project, we decided to move on with two different type of resolvers for wget2. The user has the freedom to choose the type of resolver that they want to use in order to get the IP and connect to the servers they wish for. The two revolvers are, as mentioned in the DOH(DNS-over-HTTPS) resolver and a simple system call resolver using getaddrinfo(). The second one being the default resolver for wget2. The getaddrinfo() resolver already existed for wget2 and my task here was to refactor it and bring everything under one roof. The second resolver functions are almost done and will be moving into the second part of the resolver as soon as I get a green signal from my mentor as well :D

That’s it for now, will get more of the updates as soon as possible and when I get few more interesting tweaks inside wget2 codebase as a part of my project :)

Cya all, Cheers.

Again, cheers to another fantastic summer coming up next :D.

A few days before, or better to be - 2 months before, I started to contribute towards GNU project and specifically to Wget2. To be honest, I could say that the exposure towards it and learning exposure I received by just submitting a few patches and researching about the GSoC project was tremendously huge.

By this time I hope you all understood that I got in as a student developer Intern with Google Summer of Code, again!!

This is just an introductory post, further details and others will be following this as soon as possible according to the way I would be able to accommodate my schedule. Since this summer is going to be tight as hell. (Will let you know why later :P).

Basically, my project for wget2 in GNU project is to provide support for a DNS resolver over HTTP with SSL verification, that is, DNS-over-HTTPS. For that, I would be implementing the whole DNS packet acquiring and everything from scratch since Wget2 doesn’t have support for it. On an overview, I will be implementing DNS packets, IPv4, IPv6, DNS-SD, EDNS and much more.

DNS over HTTPS is a web protocol that argues for sending DNS requests and receiving DNS responses via HTTPS connections, hence providing query confidentiality. DoH provides more than just privacy – it also helps guarantee the integrity of the response users receives to their requests. Because the DNS response is invisible between responder and user, ISPs and others in the end-to-end network chain can’t interfere in the responses.

Henceforth, we provide a plan for a new implementation of a parsing DNS over HTTPS. In the process, we would create a new library to handle DNS resolution. Further, we provided added support for handling IPv4 and IPv6 DNS packets as well as support for EDNS. The integration with HTTP provides a transport suitable for traditional DNS clients seeking access to the DNS. In the end, our client will be capable of sending DNS queries and getting DNS responses over HTTP using https:// and implies TLS security integrity and confidentiality.

Furthermore, we plan to provide support for DNS Service Discovery which is a way of using standard DNS programming interfaces, servers, and packet formats to browse the network for services.

That would be it for this time folks, will get back to you with further details and updates regularly from now onwards.

You can read the full GSoC proposal here if someone wishes to see it and if somebody is familiar with DNS-over-HTTPS implementation, please do feel free to ping me and help me out with your valuable suggestions :)

Cheers.

Happy New Year to all! The year 2017 has been a rollercoaster, to be honest. Well, it was rich and prosperous year regarding in technical terms. It was a beautiful year of great learning, splendid travel and got to network with some fantastic folks all around the globe.

The best reason for making this 2017 incredible for me is KDE. One of the exciting community I have ever seen! It all started at the end of 2016, I got intrigued by the Tagline of KDE, “Experience freedom”. I started contributing to various projects inside KDE. My initial start was with Konsole, system settings, KIO and various educational suite programs. Moving on, I came across a student program organized by KDE named KDE-SoK and I was selected for it, yay!!!

The project was with KStars(KDE’s amateur astronomy software which provides real-time and an accurate graphical simulation of the night sky, from any location on Earth.) to collect a new set of images from NASA/ESO catalogs along with orientation and pixel scale (arcsecs/pixel) from the whole set of Messier Catalog (which is a collection of 110 astronomy objects in the night sky). Images were processed for overlay in KStars using OpenCV, so to have transparency and to modulate according to the software.

After the successful completion of the project, I was invited as a speaker to give a talk on “Object-Tracking Using OpenCV and Qt” at KDE India Conference held at IIT Guwahati in March 2017. Again, yay!! :P. There I met a lot of enthusiastic KDE people out there. :)

Later on, I was selected as a Google Summer of Code (2017 edition) intern funded by Google. I contributed to Krita(Krita is a professional and open source painting program) of KDE. The project was to Integrate share.krita.org.

Share.krita.org is a place where users can share Krita scripts, images, brush packs and more. This project has two parts: integrate with the KNSCore part of the KNewStuff framework (or reimplement the protocol) and create a GUI for sharing. The second part is improving the support for creating and editing bundles. Bundles can contain brushes, patterns, gradients and so on. Also, Krita needed basic support for creating and editing bundles.

Thus said, the entire journey with KDE last year was splendid!! The community as such is a group of awesome and enthusiastic people from all around the globe. To do more than I could last year – I hope to be a part of this great venture in 2018.

Here’s to a productive year, and looking forward to the same in 2018. You can see more of what happened in KDE in 2017 on the KDE year-end fundraiser page. Please do show some of your support to the community and its developers through the fundraiser to built your favorite software :)

Some of my blog post regarding my work last year:

• KDE SoK 2016-2017
• Introduction for GSoC 2017 with KDE, Krita
• Awesome::KDE Community :)
• GSoC ’17 – Week #1
• GSoC ’17 – Week #2
• GSoC ’17 – Week #3
• GSoC ’17 – Week #4
• GSoC ’17 – Week #5
• GSoC’17 - Week #6
• GSoC’17: From 1 Line of Code to 4828

To the awesome year coming right your way,

Cheers.

For me, if you asked what was FOSS club before 6 months, I would have said that it’s just a club which deals with something related to coding about free and open source software that’s it, nothing else. But now after 6 to 7 months roughly I would say for me FOSS club is everything. The lab, the people, the energy and the enthusiasm with which the club move forward make me feel this as my second home. When most of my batch mate leaves home I stay here itself in college and do something worthwhile. Yeah! Most of them asked me why don’t you go home? I would rather smile at them than answering to that question. I say in my mind that yeah, I do miss my family and friends, but the aim to be better and make my family proud drives me to stay here and learn. But, most of that feeling of missing my family was gone as I had a new family up here, Our own FOSS family.

Now after just 6 months, if again it’s asked that what is FOSS for you, I would say it’s an Emotion. Being a part of this really cool community over here in FOSS@Amrita, Whatever I did here maybe it can be called as achievements, but it feels certainly something else, a feeling that keeps you to move on and learn again new stuff. From learning new technologies to choosing an open source organization to as KDE to contribute, again learning new technologies like Qt with C++ and getting selected for KDE SoK and writing multiple articles for open source magazines and much more would only have happened with a club and mentors like these here. If it wasn’t for mentors like Vipin sir and seniors like Harish ettan, chirath ettan, and anu chechi and many others (Taking all their names would bring a list up here.) to motivate me when I was down and help me while I was stuck and to bring me up to try again harder to achieve what was necessary and chose what to do and not to.

After joining this club and the KDE community I became so socialized as at first I never used to interact with many people, but now it seems like I am changing from an introvert to an extrovert, even if my whole body seems to break down during that process, but the thought that members of this club will be there for me every time, helps me to do whatever I wished for.

After all, I wouldn’t be in this position (at least I think) without the help of FOSS club and it’s members. I would like to thank each and everyone here in FOSS@Amrita who helped me to be what I acquired within this time span always remembering that, without you, I wouldn’t be capable of doing what all I have done.

So, to build an application, you can either follow the instructions given in the README or INSTALL files inside the repository. Usually, this involves the following steps:

~$ cd "your-source-code-directory"
~$ mkdir build cd build cmake .. (don't forget the dots! )
~$ make -j"no of processors -1" (eg: make -j5) ​
~$ make install

​What happens when you follow these set of commands is that ​your application is compiled and the build files, i.e the output of the compilation stage are left inside the build folder you just created.

Now in the last step, i.e when you run “make install”, all your build output is put in the right place, so as to get picked up at the time you launch your application. This is by default in /usr/bin or so.

So when any file needs to be moved to / prefix, it requires you to authenticate with “sudo” and it’s not advisable to install any application to a / prefix.

If you haven’t followed the above, don’t worry. Just remember, try NOT to use “sudo” in any commands you might use while building your applications.

​So what is the workaround for this? Whatever you build needs to be put(or installed) to a place from where your system can pick it up on runtime(i.e when you launch any app).​ For this purpose, it is suggested that you always install any application which you build from source to a custom prefix. In my case, development environment looks like this:

so when you configure your prefix properly, instead of installing to /usr, your build outputs would be installed to ~/devel/install/ and for this, you do not need to use “sudo” anywhere. The whole idea is that if something were to go wrong with an executable or if while installing, some configurations get altered your system wide configurations shouldn’t get affected(i.e configs in / prefix).

​​So in my case, I intend to install to ~/devel/install and I clone my source codes inside ~/devel/src folder. Once you have this directory structure, to make things simple for you, add this in bashrc or zshrc according to which bash you are using.

export KDE_SRC=/home/aniketh/devel/src 
export KDE_BUILD=/home/aniketh/devel/build 
export KDE_INSTALL=/home/aniketh/devel/install 
#export QTDIR=/usr export KF5=$KDE_INSTALL
export PATH=$KF5/bin:$PATH export XDG_DATA_DIRS=$KF5/share:$XDG_DATA_DIRS 
export XDG_CONFIG_DIRS=$KF5/etc/xdg:$XDG_CONFIG_DIRS:/etc/xdg:/usr/local/etc/xdg 
export QT_PLUGIN_PATH=$KF5/lib64/plugins:/usr/local/lib64/plugins:$QT_PLUGIN_PATH 
export QML2_IMPORT_PATH=$KF5/lib64/qml:$KF5/lib/x86_64-linux-gnu/qml:/usr/local/lib64/qml:$QML2_IMPORT_PATH 
export QML_IMPORT_PATH=$QML2_IMPORT_PATH:$QML_IMPORT_PATH 
export CMAKE_PREFIX_PATH=$KF5:$CMAKE_PREFIX_PATH 
export KDEDIRS=$KDE_INSTALL:$KDEDIRS export LD_LIBRARY_PATH=$KDE_INSTALL/lib64:$LD_LIBRARY_PATH 
export PKG_CONFIG_PATH=/usr/lib/pkgconfig:$KDE_INSTALL/lib64/pkgconfig:/usr/share/pkgconfig:$PKG_CONFIG_PATH 
export KDE_INTEGRATION="true"

You just need to open the terminal and write down.

~$ vim .zshrc or .bashrc

You can use any of the text editors and add the variables.

So, the obvious thing to do here would be to add a command to your .bashrc or .zshrc which would source the environment variables in your script on every session’s startup. (This is what we have done, instead of writing down the whole set of the environment variable in a file and source it every time. So rather doing this we have initialised it once and no more we have to worry about that).

Once you’ve done this, to build any application, you will need to do just the following:

cmake -DCMAKE_INSTALL_PREFIX=$KDE_INSTALL .. 
make -j"no_of_processors" 
make install

That’s it really. When we run the cmake command, it is obvious that we would encounter with error messages saying that some API’s needed for the project is missing, hence we will have to find the package which is missing.

The best way to do that is to give a search in the apt cache.

~$ apt-cache search package_name

After that run the cmake again till you get the configuration is done.

Hence that it, we will get the project up and running. If you encounter with other problems please feel free to contact me :)

Cheers.