Skip to content
This repository was archived by the owner on Mar 8, 2020. It is now read-only.

getting-started.md

Latest commit

 

History

History
237 lines (167 loc) · 10.3 KB

getting-started.md

File metadata and controls

237 lines (167 loc) · 10.3 KB
Raw

Getting Started

Using the online web client

The easiest way to get started with Babelfish is to try the online playground where you can write or paste your code and run the parser to see the generated UAST.

Installing bblfshd locally

After playing with the web client, you will probably want to get Babelfish running locally. The first thing to do for that is to setup and run the bblfshd command. Once the server is running, you can connect to it using any of the available clients.

Prerequisites

Running with Docker (recommended)

The easiest way to run the bblfshd daemon is using Docker. With respect to privileges of bblfshd daemon, there are two ways of executing it:

  • As rootless container. You will need to get the file bblfshd-seccomp.json. Further requirements to run in rootless mode are available at bblfshd#rootless.md. Examples below use this mode.
  • In privileged mode (we advise against it). Check out instructions here.

With respect to storage, you can run it in one of these modes:

  • Embedded drivers mode, meaning that recommended drivers will be included in the Docker image. Installing more drivers is similar to stateless mode. The Docker image will weight more of course, and there may be language drivers included in it you will not use.
  • Stateless mode, meaning bblfshd is fethed without any driver included, and the drivers you install in it will be wiped out once you remove the container.
  • Stateful mode, using a Docker volume to store part of the container internal filesystem and thus add persistence. bblfhsd is fetched without drivers, as in stateless mode, but all drivers you install will persist in the volume you use, and could be plugged in another bblfshd instance or even the same if you stop bblfshd and start it again.

Embedded drivers

For embedded drivers mode, run the following command:

$ docker run -d --name bblfshd \
  -p 9432:9432 \
  -v /proc:/newproc \
  --security-opt seccomp=./bblfshd-seccomp.json \
  bblfsh/bblfshd:latest-drivers

Stateless mode

For stateless mode, run the following command:

$ docker run -d --name bblfshd \
  -p 9432:9432 \
  -v /proc:/newproc \
  --security-opt seccomp=./bblfshd-seccomp.json \
  bblfsh/bblfshd:latest

Stateful mode

This is an alternative to stateless mode. Depending on your operating system, you can use a Docker volume or a normal local filesystem directory.

Docker volume (Linux and macOS)

First, create a volume with the command:

$ docker volume create bblfshd-cache

Then you can run the daemon with this command:

$ docker run -d --name bblfshd \
  -p 9432:9432 \
  -v /proc:/newproc \
  -v bblfshd-cache:/var/lib/bblfshd \
  --security-opt seccomp=./bblfshd-seccomp.json \
  bblfsh/bblfshd:latest
Volume mapped to a local directory (Linux only)

In this case, just specify the local directory in the -v parameter when running the daemon:

$ docker run -d --name bblfshd \
  -p 9432:9432 \
  -v /proc:/newproc \
  -v /var/lib/bblfshd:/var/lib/bblfshd bblfsh/bblfshd:latest \
  --security-opt seccomp=./bblfshd-seccomp.json \
  bblfsh/bblfshd:latest

Notes on command line parameters to "docker run"

  • The flag -d makes the container run as a daemon. You can omit it and you will see the log outputs.
  • --name bblfshd gives a name to the container as bblfshd.
  • Exposing the port (9432) with -p 9432:9432 makes it easier to connect to the gRPC server from outside the container.
  • -v /proc:/newproc is mandatory to run in rootless mode, due to a bug in libcontainer, the container technology used in bblfshd.
  • --security-opt seccomp=./bblfshd-seccomp.json allows the default syscalls Docker permits plus mount, unshare, pivot_root, keyctl, umount2 and sethostname, needed to spawn containers inside bblfshd. More on that matter can be consulted in Docker docs.

If you are behind an HTTP or HTTPS proxy server, for example in corporate settings, you will need to add the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables in the docker run command to configure HTTP or HTTPS proxy behavior.

$ docker run -d --name bblfshd \
  -p 9432:9432 \
  -v /proc:/newproc \
  -v bblfshd-cache:/var/lib/bblfshd \
  --security-opt seccomp=./bblfshd-seccomp.json \
  -e HTTP_PROXY="http://proxy.example.com:80/" \
  bblfsh/bblfshd:latest

If your system uses SELinux (like Fedora, Red Hat or CentOS among other Linux distributions) you'll need to install a policy module. You can find instructions about it in the bblfshd README.

Testing the daemon

If everything worked, docker logs bblfshd should output something like this:

[2019-05-01T17:58:29Z]  INFO bblfshd version: v2.12.1 (build: 2019-04-10T16:05:44+0000)
[2019-05-01T17:58:29Z]  INFO initializing runtime at /var/lib/bblfshd
[2019-05-01T17:58:29Z]  INFO server listening in 0.0.0.0:9432 (tcp)
[2019-05-01T17:58:29Z]  INFO control server listening in /var/run/bblfshctl.sock (unix)

Driver management

Now we need to install the driver images into the daemon. You can install the official images with the following command:

$ docker exec -it bblfshd bblfshctl driver install --recommended

If you want to install a driver for a single language, you can do so by specifying the driver URI to bblfshctl:

docker exec -it bblfshd bblfshctl driver install docker://bblfsh/python-driver:latest

You can check the installed versions executing:

$ docker exec -it bblfshd bblfshctl driver list
+------------+------------------------------------------+---------+--------+----------+--------+-------------+----------------------+
|  LANGUAGE  |                  IMAGE                   | VERSION | STATUS | CREATED  |   OS   |     GO      |        NATIVE        |
+------------+------------------------------------------+---------+--------+----------+--------+-------------+----------------------+
| python     | docker://bblfsh/python-driver:latest     | v2.3.0  | beta   | 8 months | alpine | 1.10-alpine | python:3.6-alpine    |
| java       | docker://bblfsh/java-driver:latest       | v2.2.0  | beta   | 8 months | alpine | 1.10-alpine | openjdk:8-jre-alpine |
+------------+------------------------------------------+---------+--------+----------+--------+-------------+----------------------+

You can remove a driver (e.g. python one) with:

$ docker exec -it bblfshd bblfshctl driver remove python

or all drivers with:

$ docker exec -it bblfshd bblfshctl driver remove --all
Testing the drivers

To test the driver you can execute a parse request to the server with the bblfshctl parse command, and an example contained in the Docker image:

$ docker exec -it bblfshd bblfshctl parse /opt/bblfsh/etc/examples/python.py

Use this only for testing the installation; if you want to do any real parsing with files in your local filesystem you should use one of the clients which would also allow you to run XPath queries over the results.

Running standalone

A standalone distribution of bblfshd and bblfshctl can be found at the GitHub release page. This will contain a single binary that can be run anywhere but it depends on the ostree library as explained in the readme.

bblfshd is only provided for Linux distributions, since it relies on Linux containers to run language drivers. And bblfshctl can be found for Windows, macOS and Linux.

Using bblfshctl

The bblfshd daemon comes with a command line tool called bblfshctl, which can be used to monitor and manage the daemon.

If you are using the Docker image then the command line tool is provided with the bblfsh/bblfshd image and can be used with a docker exec command.

$ docker exec -it bblfshd bblfshctl --help
Usage:
  bblfshctl [OPTIONS] <command>

Help Options:
  -h, --help  Show this help message

Available commands:
  driver     Manage drivers: install, remove and list
  instances  List the driver instances running on the daemon
  parse      Parse a file and prints the UAST or AST
  status     List all the pools of driver instances running

Note that the --help command works at any sublevel:

docker exec -it bblfshd bblfshctl driver --help

Driver management

The bblfshd's drivers can be installed, updated and removed with the driver command and its subcommands. Remember to prefix all these commands with docker exec -it bblfshd if you are running bblfshd from a Docker image.

Installing all official drivers:

$ bblfshctl driver install --recommended

Replacing an installed driver with a specific version:

$ bblfshctl driver install python bblfsh/python-driver:v2.3.0 --update

Listing all available drivers:

$ bblfshctl driver list
+------------+------------------------------------------+---------+--------+----------+--------+-------------+----------------------+
|  LANGUAGE  |                  IMAGE                   | VERSION | STATUS | CREATED  |   OS   |     GO      |        NATIVE        |
+------------+------------------------------------------+---------+--------+----------+--------+-------------+----------------------+
| python     | docker://bblfsh/python-driver:latest     | v2.3.0  | beta   | 8 months | alpine | 1.10-alpine | python:3.6-alpine    |
| java       | docker://bblfsh/java-driver:latest       | v2.2.0  | beta   | 8 months | alpine | 1.10-alpine | openjdk:8-jre-alpine |
+------------+------------------------------------------+---------+--------+----------+--------+-------------+----------------------+