1. To start the Docker container, run script/console : This will land you inside the Docker container, starting in the /src directory. You can detach from/attach to this container to pause/continue your work.

For more about the data, see Data Details below, as well as this notebook.

If you have run the setup steps above you will already have the data, and nothing more needs to be done. The data will be available in the /resources/data folder of this repository, with this directory structure.

Data is stored in jsonlines format. Each line in the uncompressed file represents one example (usually a function with an associated comment). A prettified example of one row is illustrated below.

• repo: the owner/repo
• path: the full path to the original file
• func_name: the function or method name
• original_string: the raw string before tokenization or parsing
• language: the programming language
• code: the part of the original_string that is code
• code_tokens: tokenized version of code
• docstring: the top-level comment or docstring, if it exists in the original string
• docstring_tokens: tokenized version of docstring
• sha: this field is not being used [TODO: add note on where this comes from?]
• partition: a flag indicating what partition this datum belongs to of This is not used by the model. Instead we rely on directory structure to denote the partition of the data.
• url: the url for the code snippet including the line numbers

Code, comments, and docstrings are extracted in a language-specific manner, removing artifacts of that language.

Summary statistics such as row counts and token length histograms can be found in this notebook

Downloading Data from S3

The shell script /script/setup will automatically download these files into the /resources/data directory. Here are the links to the relevant files for visibility:

The s3 links follow this pattern:

For example, the link for the java is:

The size of the dataset is approximately 20 GB. The various files and the directory structure are explained here.

Human Relevance Judgements

To train neural models with a large dataset we use the documentation comments (e.g. docstrings) as a proxy. For evaluation (and the leaderboard), we collected human relevance judgements of pairs of realistic-looking natural language queries and code snippets. Now that the challenge has been concluded, we provide the data here as a .csv , with the following fields:

• Language: The programming language of the snippet.
• Query: The natural language query
• GitHubUrl: The URL of the target snippet. This matches the URL key in the data (see here).
• Relevance: the 0-3 human relevance judgement, where «3» is the highest score (very relevant) and «0» is the lowest (irrelevant).
• Notes: a free-text field with notes that annotators optionally provided.

Running Our Baseline Model

We encourage you to reproduce and extend these models, though most variants take several hours to train (and some take more than 24 hours on an AWS P3-V100 instance).

Our baseline models ingest a parallel corpus of ( comments , code ) and learn to retrieve a code snippet given a natural language query. Specifically, comments are top-level function and method comments (e.g. docstrings in Python), and code is an entire function or method. Throughout this repo, we refer to the terms docstring and query interchangeably.

The query has a single encoder, whereas each programming language has its own encoder. The available encoders are Neural-Bag-Of-Words, RNN, 1D-CNN, Self-Attention (BERT), and a 1D-CNN+Self-Attention Hybrid.

The diagram below illustrates the general architecture of our baseline models:

This step assumes that you have a suitable Nvidia-GPU with Cuda v9.0 installed. We used AWS P3-V100 instances (a p3.2xlarge is sufficient).

Start the model run environment by running script/console :

This will drop you into the shell of a Docker container with all necessary dependencies installed, including the code in this repository, along with data that you downloaded earlier. By default, you will be placed in the src/ folder of this GitHub repository. From here you can execute commands to run the model.

Set up W&B (free for open source projects) per the instructions below if you would like to share your results on the community benchmark. This is optional but highly recommended.

The entry point to this model is src/train.py . You can see various options by executing the following command:

To test if everything is working on a small dataset, you can run the following command:

Now you are prepared for a full training run. Example commands to kick off training runs:

Training a neural-bag-of-words model on all languages

The above command will assume default values for the location(s) of the training data and a destination where you would like to save the output model. The default location for training data is specified in /src/data_dirs_.txt . These files each contain a list of paths where data for the corresponding partition exists. If more than one path specified (separated by a newline), the data from all the paths will be concatenated together. For example, this is the content of src/data_dirs_train.txt :

By default, models are saved in the resources/saved_models folder of this repository.

Training a 1D-CNN model on Python data only:

The above command overrides the default locations for saving the model to trained_models and also overrides the source of the train, validation, and test sets.

Options for —model are currently listed in src/model_restore_helper.get_model_class_from_name .

Hyperparameters are specific to the respective model/encoder classes. A simple trick to discover them is to kick off a run without specifying hyperparameter choices, as that will print a list of all used hyperparameters with their default values (in JSON format).

We are using a community benchmark for this project to encourage collaboration and improve reproducibility. It is hosted by Weights & Biases (W&B), which is free for open source projects. Our entries in the benchmark link to detailed logs of our training and evaluation metrics, as well as model artifacts, and we encourage other participants to provide as much detail as possible.

We invite the community to submit their runs to this benchmark to facilitate transparency by following these instructions.

How to Contribute

We anticipate that the community will design custom architectures and use frameworks other than Tensorflow. Furthermore, we anticipate that additional datasets will be useful. It is not our intention to integrate these models, approaches, and datasets into this repository as a superset of all available ideas. Rather, we intend to maintain the baseline models and links to the data in this repository as a central place of reference. We are accepting PRs that update the documentation, link to your project(s) with improved benchmarks, fix bugs, or make minor improvements to the code. Here are more specific guidelines for contributing to this repository; note particularly our Code of Conduct. Please open an issue if you are unsure of the best course of action.

To initialize W&B:

Navigate to the /src directory in this repository.

If it’s your first time using W&B on a machine, you will need to log in:

You will be asked for your API key, which appears on your W&B profile settings page.

The licenses for source code used as data for this project are provided with the data download for each language in _licenses.pkl files.

This code and documentation for this project are released under the MIT License.

About

Datasets, tools, and benchmarks for representation learning of code.