Demo Virtual Machine (VM)

A conveniently packaged virtual machine (VM) preinstalled with dA Platform for demonstration purposes.

Prerequisites

Before we begin, in order to run the demo VM you will need the following prerequisites:

  • Working installation of VirtualBox
  • Computer with at least 6GB of available RAM and 2 CPU cores

Downloading

Visit the product downloads page and download the linked VM image.

Installation

Navigate to the file you downloaded in the previous step, and double click the OVA (Open Virtualization Appliance) file. VirtualBox should then open up and prompt you to import the appliance.

If for some reason VirtualBox doesn’t prompt you, then you’ll need to manually initiate the import. To do this, open VirtualBox and from the menu select File > Import Appliance, then select the OVA file you downloaded earlier.

You’ll then be shown a quick overview of the “Appliance settings” for review before importing the appliance, which you can proceed with by clicking on the Import button.

Last of all a license prompt will appear, which contains the licensing agreement for the VM. You must accept before you can proceed and import the appliance. Once you’ve clicked Accept, the VM import will begin.

Running

After the import has finished, you can start the VM by opening VirtualBox and clicking Start on the imported VM, which will be named daplatform-demo-.*

Getting Started

Note

The VM requires a certain amount of resources (6GM of RAM and 2 CPU cores minimum) to run smoothly, and the more resources you can provide it the better. Please keep this in mind before attempting to run any non-trivial Flink jobs, as you may find that the VM has insufficient resources to run such jobs.

In order to get started with the VM, open the Application Manager user interface by entering daplatform/ into the address bar of Firefox, or by using the Bookmarks we’ve installed in Firefox. Then from the Application Manager UI press the Create Deployment button.

Application Manager has been set up with two example jobs:

  • fraud-detection
  • top-windowing-speed

These are basic examples intended to demonstrate deployments in Application Manager. They are deliberately started in the Cancelled state, if you wish to run them, click on the desired deployment to go to that deployments overview page, and then press the Start button.

Below we’ll take you through the steps necessary to deploy the fraud-detection example using the Application Manager user interface.

To deploy the fraud detection example from the local Minio server, fill in the form with the following parameters:

  • Name: Enter any name
  • Deployment Target: Kubernetes
  • Upgrade Strategy: Stateless
  • Initial State: Cancelled
  • Start from Savepoint: Latest
  • Paralleism: 1
  • Jar URI: http://minio/daplatform-demo/example-jobs/fraud-detection.jar
  • For JobManager and TaskManager CPU: 1

In order to start this deployment, press the Start button in the Application Manager UI.

Set up checkpointing and savepointing

One of the main features and advantages of Application Manager is the way it handles stateful streaming deployments.

In order to demonstrate this, we are going to use a sample Flink streaming job that is stateful. The URI of the Jar for this job is http://minio/daplatform-demo/example-jobs/fraud-detection.jar.

In addition to the Jar, we need to configure the following settings:

  • Entry Class: com.dataartisans.FraudDetection
  • Main Args: --fsStatePath s3://daplatform-demo/checkpoints

We also need to configure S3 for the Flink job we are going to launch. For that, please follow the instructions below.

Configure S3 with Minio

The VM also includes Minio. Minio is an Amazon S3 compatible object storage server. It has a user interface which is accessible via http://minio/, with username admin and password password.

To configure Flink to use Minio, set the following Flink configuration parameters in Application Manager:

  • s3.access-key: admin
  • s3.secret-key: password
  • s3.endpoint: http://minio
  • state.savepoints.dir: s3://daplatform-demo/savepoints

Design

The VM consists of a single node Kubernetes cluster running Application Manager and other supplemental components (Grafana, Kibana, Logstash, Minio, etc.). The default session is configured for auto-login and has the screen locker disabled, however should you somehow get logged out of the session, the user credentials are username: vagrant, password: vagrant.

VirtualBox’s Guest Additions have been pre-installed in the guest VM, which provide a number of useful desktop/host integrations:

  • Mouse pointer integration
  • Shared clipboard
  • Better video support
  • Shared folders
  • Seamless windows

All of the Kubernetes deployments are using persistent volumes within the VM, and are available under /var/lib/daplatform. If you wish to persist data from the VM you can use Shared Folders in VirtualBox to map in storage from your host system, and then copy the files out that way.

VirtualBox can auto-mount the shared folder for you if you restart the VM, but if you’d prefer to not restart the VM, you can manually mount the shared folder yourself by issuing the following command inside the VM: sudo mount -t vboxsf your-share-name-here ~/some-example-path