Merge develop into master

Author: lkrcal

.artifactignore

@@ -0,0 +1,5 @@
+.git
+.idea
+.DS_Store
+.artifactignore
+.gitignore

.gitignore

@@ -1,3 +1,4 @@
+.DS_Store
 
 # Created by https://www.gitignore.io/api/node,intellij
 # Edit at https://www.gitignore.io/?templates=node,intellij
@@ -165,6 +166,3 @@ typings/
 .dynamodb/
 
 # End of https://www.gitignore.io/api/node,intellij
-
-.DS_Store
-/iofog-agent/id_ecdsa*

README.md

@@ -1,28 +1,164 @@
 # ioFog Demo
 
-This repository orchestrates ioFog Agent, Controller, and Connector in a Docker Compose environment for the purpose of demonstrating a deployment of the ioFog stack.
+This repository demonstrates the capabilities of ioFog. It spins up the ioFog stack (Agent, Controller, and Connector) on a local machine in a Docker Compose environment. This basic ioFog stack setup constitutes a small, but fully configured Edge Compute Network (ECN).
 
-# Usage
+Optionally, this demo also creates a sample application (microservices deployed on the ioFog stack).
 
-To spin up Agent, Controller, and Connector containers within a virtual network:
-```
+This demo repository is used as supplementary materials for [ioFog quickstart](https://iofog.org/docs/1.0.0/getting-started/core-concepts.html) and [ioFog tutorial](https://iofog.org/docs/1.0.0/tutorial/introduction.html) guides. 
+
+
+# Prerequisites
+
+The ioFog demo requires one of the following systems and tools to be installed. The scripts in this demo do not install any of these tools, but they check for sufficient versions.
+
+Supported operating systems:
+
+* Linux (kernel v3.10+)
+* macOS 10.12+
+* Windows 7+
+
+Requires tools:
+
+* Docker 1.10+ ([installation instructions](https://docs.docker.com/install/))
+* Docker-compose 1.22+ ([installation instructions](https://docs.docker.com/compose/install/))
+
+
+# Try ioFog - Simple Edge Compute Network
+
+The main interaction with this demo repository is encapsulated into a set of simple scripts: `start.sh`, `test.sh` and `stop.sh`. Interactions with the ioFog components can be done using a command line interface available in all the services of the stack, or using a REST API.
+
+## Spin Up The ioFog Stack
+
+Spin up the blank ioFog stack (Agent, Controller, and Connector) on the local machine.
+
+```sh
 ./start.sh
 ```
 
-To verify the services are provisioned correctly:
-```
+Verify the iofog stack is provisioned correctly. The automated tests run a smoke test suite on the blank ioFog stack, testing basic operations.
+
+```sh
 ./test.sh
 ```
 
-To interact with Agent, Controller, and Connector you must exec into each respective container and use the CLI:
+You can also verify manually that the ioFog stack containers are correctly started.
+```sh
+docker ps --filter "name=iofog"
 ```
-docker exec -it iofog-controller /bin/bash
-iofog-controller help
+
+When you are finished, tear down the ioFog stack and all services deployed on it.
+
+```sh
+./stop.sh
 ```
 
-Note that if you would like to use Agent, Controller, and Connector's REST APIs, you will have to modify the Compose environment to deploy with network_mode "host".
+## Build from local packages
+
+If you have a local version of the Agent, Controller and Connector, you can chose to build the containers using those local packages.
+To do so, you will need a debian package (`.deb`) for the Agent and the Connector and a tarball (`.tgz`) for the Controller.
+You can provide `start.sh` with an option for each local package you want to use.
 
-When you are finished, it is recommended that you teardown:
+### Example
+Folder structure:
+```text
+* services
+* init
+* test
+* local-packages  # Example folder where you would store your local packages
+    - iofog-agent_2.0.deb
+    - iofog-connector_2.0.deb
+    - iofog-controller_2.0.tgz
+* start.sh
+* stop.sh
+* ...
 ```
+
+Command:
+```sh
+./start.sh -a ./local-packages/iofog-agent_2.0.deb -cn ./local-packages/iofog-connector_2.0.deb -ct ./local-packages/iofog-controller_2.0.tgz
+```
+
+## Force rebuild
+If you have previously built the containers using local packages, or remote packages and you want to ensure that running `start.sh` will rebuild the images, you can also provide the `--no-cache` option
+
+
+```sh
+./start.sh --no-cache
+```
+
+## ECN Status
+
+```sh
+./status.sh # Will show you all iofog-related containers currently running.
+```
+
+## Interacting With The ioFog Stack - CLI
+
+The simplest way to interact with Agent, Controller, and Connector deployed on a machine you have access to is to use the command line interface. The main interaction point for users is the Controller.
+
+```sh
+docker exec -it iofog-controller iofog-controller help
+```
+For the purpose of this demo, all ioFog components are spun up in separate Docker containers. The Controller's container is called `iofog-controller` (the first occurrence in the above command) and the executable inside the container is also called `iofog-controller` (the second occurrence).)_
+
+Names for all the containers created in the demo are `iofog-agent`, `iofog-controller` and `iofog-connector`.
+
+The initialization scripts used to setup the ioFog stack / ECN are using the CLI interface. Feel free to refer to these for more inspiration.
+
+Full reference of the CLI for all ioFog stack components is available at the ioFog website:
+
+* https://iofog.org/docs/1.0.0/controllers/cli-usage.html
+* https://iofog.org/docs/1.0.0/agents/cli-usage.html
+* https://iofog.org/docs/1.0.0/connectors/cli-usage.html
+
+## Interacting With The ioFog Stack - REST API
+
+
+Full reference of the REST API for all ioFog stack components is available at the ioFog website:
+
+* https://iofog.org/docs/1.0.0/controllers/rest-api.html
+* https://iofog.org/docs/1.0.0/agents/local-api.html
+* https://iofog.org/docs/1.0.0/connectors/api-reference.html
+
+You can try using the REST API directly on your machine with the ioFog stack running.
+```sh
+curl --url 'http://0.0.0.0:51121/api/v3/status' 
+```
+
+
+# Try ioFog - Tutorial Application Deployed On ioFog
+
+Apart from creating just the ioFog stack, we can also deploy an ioFog application on the stack. Here we demonstrate it on the tutorial application from the ioFog website.
+
+First, create all services for a tutorial ioFog application. You don't have to start the iofog stack manually, it will be created if necessary.
+
+```sh
+./start.sh tutorial
+```
+
+When you are done with the tutorial, you can tear down the sample application together with the ioFog stack. Note there is currently no wya in the demo to tear down just the tutorial application.
+```sh
 ./stop.sh
-```
+```
+
+# Structure Of This Repository
+```text
+* services                  # Service Dockerfiles and customization files
+    - iofog
+        + iofog-agent       # Agent service files - part of the iofog stack
+        + iofog-connector   # Connector service files - part of the iofog stack
+        + iofog-controller  # Controller service files - part of the iofog stack
+* init
+    - iofog                 # plain ioFog stack initialization service        
+    - tutorial              # tutorial initialization service
+* test
+    + conf                  # generated test configuration files 
+* azure-pipelines.yml
+* docker-compose-iofog.yml
+* docker-compose-tutorial.yml
+* docker-compose-test.yml
+* start.sh
+* stop.sh
+* test.sh
+* utils.sh
+```

azure-pipelines.yml

@@ -1,8 +1,9 @@
 trigger:
-- master
-- develop
-- blank-environment
-- demo-environment
+  - master
+  - develop
+pr:
+  - master
+  - develop
 
 pool:
   vmImage: 'ubuntu-16.04'
@@ -19,11 +20,6 @@ steps:
     echo $(gcp.svcacc) | docker login -u _json_key --password-stdin https://gcr.io
   displayName: 'Docker connect to Registry'
 
-- script: |
-    sed -i "s|image:.*|image: $(images.runner)|" docker-compose-test.yml
-    cat docker-compose-test.yml
-  displayName: 'Configure Test Runner image'
-
 - script: |
     ./start.sh
   displayName: 'Start Connector, Controller, and Agent'
@@ -34,4 +30,18 @@ steps:
 
 - script: |
     ./stop.sh
-  displayName: 'Stop Connector, Controller, and Agent'
+  displayName: 'Stop Connector, Controller, and Agent'
+
+- script: |
+    tar -c --transform 's,^\.,demo,' --exclude-from=.artifactignore -v --bzip2 -f $BUILD_ARTIFACTSTAGINGDIRECTORY/demo.tar.bz2 .
+
+- task: PublishBuildArtifacts@1
+  inputs:
+    pathtoPublish: '$(Build.ArtifactStagingDirectory)/demo.tar.bz2'
+    artifactName: 'demo.$(Build.BuildId).tar.bz2'
+
+- script: |
+    echo "===== IOFOG AGENT LOG ====="
+    docker exec iofog-agent cat /var/log/agent.out.log
+  displayName: 'Print logs'
+  condition: failed()

conf/.gitignore

@@ -0,0 +1,60 @@
+version: "3"
+services:
+    iofog-connector:
+        image: iofog-connector:local
+        build:
+            context: ./services/iofog/iofog-connector
+            args:
+                - LOCAL_CONNECTOR_PACKAGE
+        container_name: iofog-connector
+        ports:
+            - "53321:8080"
+        network_mode: bridge
+
+    iofog-controller:
+        image: iofog-controller:local
+        build: 
+            context: ./services/iofog/iofog-controller
+            args:
+                - LOCAL_CONTROLLER_PACKAGE
+        container_name: iofog-controller
+        depends_on:
+            - iofog-connector
+        ports:
+            - "51121:51121"
+        environment:
+            - NODE_ENV=development
+        links:
+            - iofog-connector:iofog-connector
+        network_mode: bridge
+
+    iofog-agent:
+        image: iofog-agent:local
+        build: 
+            context: ./services/iofog/iofog-agent
+            args:
+                - LOCAL_AGENT_PACKAGE
+        depends_on:
+            - iofog-controller
+        privileged: true
+        volumes:
+            - /var/run/docker.sock:/var/run/docker.sock
+        container_name: iofog-agent
+        ports:
+            - "8081:22"
+            - "54321:54321"
+        links:
+            - iofog-controller:iofog-controller
+        network_mode: bridge
+
+    iofog-init:
+        build: ./init/iofog
+        container_name: iofog-init
+        depends_on:
+            - iofog-controller
+        volumes:
+            - /var/run/docker.sock:/var/run/docker.sock
+        links:
+            - iofog-controller:iofog-controller
+            - iofog-connector:iofog-connector
+        network_mode: bridge

docker-compose-iofog.yml

@@ -1,15 +1,11 @@
-services:
-  test-runner:
-    container_name: test-runner
-    environment:
-    - LOCAL=1
-    image: iofog/test-runner:latest
-    networks:
-    - iofog
-    volumes:
-    - ./conf:/conf
 version: '3'
-volumes:
-  conf: null
-networks:
-  iofog: null
+services:
+    # Test Harness image
+    test-runner:
+        container_name: test-runner
+        environment:
+            - LOCAL=1
+        image: iofog/test-runner:latest
+        volumes:
+            - ./test/conf:/conf
+        network_mode: bridge

docker-compose-test.yml

@@ -0,0 +1,12 @@
+version: "3"
+services:
+    tutorial-init:
+        build: ./init/tutorial
+        container_name: tutorial-init
+        depends_on:
+            - iofog-controller
+        volumes:
+            - /var/run/docker.sock:/var/run/docker.sock
+        links:
+            - iofog-controller:iofog-controller
+        network_mode: bridge

docker-compose-tutorial.yml

@@ -0,0 +1,7 @@
+FROM docker:dind
+
+RUN apk add --no-cache curl jq
+
+ADD ./provision.sh /
+
+CMD ["sh", "-c", "/provision.sh"]